mediawiki_dump_tools.git
3 years agoadd tests for wikipedia, malformed xml, bzip2, correct bz2 bug in wikiq.
groceryheist [Wed, 4 Jul 2018 22:08:30 +0000 (15:08 -0700)]
add tests for wikipedia, malformed xml, bzip2, correct bz2 bug in wikiq.

3 years agocreate baseline tests for xml dump processing
groceryheist [Wed, 4 Jul 2018 06:43:47 +0000 (23:43 -0700)]
create baseline tests for xml dump processing

3 years agoa number of small updates and fixes
Benjamin Mako Hill [Thu, 17 May 2018 21:37:20 +0000 (14:37 -0700)]
a number of small updates and fixes

- fix regex for filename/filetype matches
- unload all files not just ones with end with xml in 7z archives
- fix bug that broke stdout
- minor cosmetic fixes
- updated mediawiki-utilities submodule to latest version

3 years agosupport 7z archives with multiple files. add urlencode paraeter
groceryheist [Thu, 7 Dec 2017 23:10:56 +0000 (15:10 -0800)]
support 7z archives with multiple files. add urlencode paraeter

4 years agofix code to work with bzip files
Benjamin Mako Hill [Tue, 7 Feb 2017 02:25:17 +0000 (18:25 -0800)]
fix code to work with bzip files

6 years agoadded list of compressed dump files to .gitignore
Benjamin Mako Hill [Thu, 23 Jul 2015 19:16:31 +0000 (12:16 -0700)]
added list of compressed dump files to .gitignore

6 years agoadded support to parse namespaces from title
Benjamin Mako Hill [Thu, 23 Jul 2015 19:12:20 +0000 (12:12 -0700)]
added support to parse namespaces from title

This is necessary for wikis (e.g., Wikia XML dumps) that do not include
namespace metadata as tags within each <page>.

6 years agoadded README file to document the submodule
Benjamin Mako Hill [Thu, 23 Jul 2015 02:55:08 +0000 (19:55 -0700)]
added README file to document the submodule

6 years agocreated new repository for wikiq with Mediawiki-Utilities as a submodule
Benjamin Mako Hill [Thu, 23 Jul 2015 02:44:52 +0000 (19:44 -0700)]
created new repository for wikiq with Mediawiki-Utilities as a submodule

Community Data Science Collective || Want to submit a patch?