2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* configure.ac: rely on uctodata 0.4

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* : citation fix

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* : layout fix rendered

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto_manual.tex: layout fix

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto_manual.tex: no need for ILK affiliation anymore in
	documentation #2

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto_manual.tex: Updates to ucto manual for 0.9.6
	(installation procedure, new language codes, contact address). Text
	still need to be revised/expanded sometime. #2

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto.1: man page update

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: updated the readme for the new language codes

2017-01-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am, src/Makefile.am: unstall and look for
	datafiles in $PREFIX/share/ucto

2017-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: added more DATE testcases
	and the desired result

2017-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : Merge pull request #15 from sanmai-NL/static_analysis Check code quality during CI

2017-01-15  Sander Maijers <S.N.Maijers@gmail.com>

	* .travis.yml: Check code quality during CI Use CppCheck and the Clang Static Analyzer and fail on them by
	default.

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small refactoring to avoid potential 0 pointer

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: removed unused variable

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: minor refactoring

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: removed dead code

2017-01-16  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: Temper expectations about ucto's simple date/time
	recognising abilities (#16)

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/Makefile.am, tests/testoption2, tests/testtwitter: fixed
	test scripts to correctly find the uctodata

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/es.abr, config/exotic-eos.eos, config/exotic-quotes.quote,
	config/ligatures.filter, config/nl_afk.abr, config/pt.abr,
	config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-it,
	config/tokconfig-nl, config/tokconfig-nl-sonarchat,
	config/tokconfig-nl-twitter, config/tokconfig-nl-withplaceholder,
	config/tokconfig-pt, config/tokconfig-ru, config/tokconfig-sv,
	config/tokconfig-tr: removed files that were moved to uctodata long
	ago

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: typo

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: updated NEWS for the release

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: We do not longer require the uctodata package to be
	installed. But issue a notice! If present we check for a recent and
	decent version.

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 3bb3f7b6fba6a1d1ce566591cba65b606bbf738b Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Fri Jan 6 13:10:35 2017
	+0100

2017-01-06  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-generic: Updated tokconfig-generic with version
	information

2017-01-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-01-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: updatede NEWS for upcoming release

2016-12-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/unicode.cxx: some refactoring, generally use
	back() to get last element of a collection

2016-12-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h, include/ucto/textcat.h,
	include/ucto/tokenize.h, include/ucto/unicode.h, src/setting.cxx,
	src/textcat.cxx, src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 
	also entering 2017 now

2016-11-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: added libexttextcat to the travis dependencies

2016-11-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fix merge problem

2016-11-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit b9af1264b2d4ad9f96ce59afd82ab4ca8c8e29c0 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Nov 21 10:42:32 2016
	+0100

2016-11-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: ok, no more exit() calls

2016-11-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: better error messages (including filename)

2016-11-14  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto.1: fixed typo in manpage

2016-11-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: fiddling with configure to test libexttextcat in the
	searchpath

2016-11-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: merges fix from language-detect branch. (clearer
	warning)

2016-11-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: better warninin message iin case off missing
	version number

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added a 'real' set url for language

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixes for language set/class

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testlang.ok: small fix.We should sort out
	"none" vs. "default"

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, tests/folia-lang.xml,
	tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testlang, tests/testslash.ok: added code to add correct
	language annotations. Stila bit clumsy.  Needs cleaning and
	thinking...

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/empty_line.txt, tests/testpassthru,
	tests/testpassthru.ok: merged the passthru fix from language-detect
	branch

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/{empty_line.nl.txt => empty_line.txt}, tests/testpassthru: 
	some file renaming in test

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/empty_line.nl.txt: added testfile

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testpassthru, tests/testpassthru.ok: fixed
	issue 11 (https://github.com/LanguageMachines/ucto/issues/11). added
	a test.

2016-11-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/unicode.h: added a missing include

2016-11-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/textcat.h, include/ucto/tokenize.h, src/textcat.cxx,
	src/tokenize.cxx: added const qualifiers. made sure config.h is NOT
	included in other .h files.

2016-11-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/Makefile.am, include/ucto/textcat.h,
	include/ucto/tokenize.h, src/Makefile.am, src/textcat.cxx,
	src/tokenize.cxx, src/ucto.cxx: put all textcat related stuff in own
	files

2016-11-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: 3 typos corrected in usage()

2016-11-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: 3 typos in usage() corrected

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/textcat.cfg, tests/multilang.txt: new files for testing

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: updated test

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fix detection

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/testlang: add langauage detection for NON folia input. still
	buggy

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: changed testresult

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am, include/ucto/setting.h,
	include/ucto/tokenize.h, src/setting.cxx, src/tokenize.cxx,
	tests/testlang: next step in language detection. still rough edges

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: started incorporationg
	textcat stuff. Now working/used yet

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: added textcat check to configure

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small fix

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added a stub for
	upcoming language support

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fiddle with the (default) language settings

2016-10-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: working
	on language detection.

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx: more cleanup

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/setting.cxx, src/tokenize.cxx: 
	cleanup

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* bootstrap.sh, include/ucto/setting.h, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/setting.cxx, src/tokenize.cxx,
	src/unicode.cxx: more code reshuffling. keep everything in one
	namespace for now

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/Makefile.am, include/ucto/setting.h,
	include/ucto/tokenize.h, include/ucto/unicode.h, src/Makefile.am,
	src/setting.cxx, src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 
	some refactoring. splitup of huge tokenize.cxx file

2016-10-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: some
	small refactoring

2016-10-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: save
	current work. all tests still work

2016-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: save
	current state. lot of new but unused code :)

2016-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: started implementing language detection

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfiles1.ok: adapted another test

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx, tests/testlang.ok: added backward compatability for
	ISO 639-1. Adapted test

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia-lang.xml: converted to ISO 639-3

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, src/ucto.cxx: simplified language parameter parsing.

2016-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: fix some language isssues

2016-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: needed for testing

2016-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/folia-lang.xml, tests/testall, tests/testlang: added language
	support for folia documents. Some ISO-639 hackery to support older
	files too

2016-10-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testinclude.ok, tests/tst.cfg: fixed
	stupid bug. 'make check' now works agaiin. And tests/testinclude now
	gives correct result

2016-09-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bump version after release

2016-09-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS for the release

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: now request at least ICU 5.2

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: modified test.py so that ucto finds the data
	automagicly, not using -c.

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to let travis run Trusty

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testall: fix exit status stuff

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testall: make sure that the python script
	returns an exit code on failure

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: one more down :{

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: ok, try icu 4.9 because Travis uses old versions

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, src/tokenize.cxx: ask for ICU 5.0 or more. assuming
	the (?i) bug is fixed then

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/small.cfg, tests/testinclude.ok: fixed test now we check
	versions

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: check version info if
	present.

2016-09-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 39ef5df2ab8f6b20eb5e43141f2897d0326b81ee Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Aug 23 16:52:28 2016
	+0200

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.en.tok.V, tests/test.fr.tok.V: adapted to newer
	insigths

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: some refactoring

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/ligaturen.nl.tok.V, tests/normalisation.nl.tok.V,
	tests/normalisationUTF16.nl.tok.V, tests/nu.nl.tok.V,
	tests/ocr.nl.tok.V, tests/testfolia.ok, tests/testslash.ok: adapted
	tests to newest insights

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: introduced type_symbol

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: added some currency tests

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added code to parse [CURRENCY]. That entry was
	silently ignored!

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/bug0054.nl.tok.V, tests/issue01.nl.tok.V,
	tests/ligaturen.nl.tok.V, tests/nu.nl.tok.V, tests/ocr.nl.tok.V,
	tests/smileys.nl.tok.V, tests/test.es.tok.V, tests/test.fr.tok.V,
	tests/testquotes.ok, tests/testquotes2.ok, tests/testslash.ok: 
	reflect advances in NOSPACE handling

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed NOSPACE in sequences

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testone: merge good idea from 'recursive'
	branch

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testpunctuation.ok: progress

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/ligaturen.nl.tok.V: another improvement

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testencoding2.ok: a moving target...

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/nu.nl.tok.V, tests/test.nl.tok.V, tests/test.py,
	tests/testencoding2.ok, tests/testone: save latest and greatest
	results

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/bug0083.nl.tok.V, tests/testencoding2.ok: results adapted to
	newest state of affairs

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: some fixes. getting closer to 'old' ucto

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: improved error message
	about invalid regex

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/abbreviations.nl.tok.V, tests/test.nl.tok.V,
	tests/testpunctfilter.ok, tests/testpunctuation.ok: adapted test
	results to improved tokconfig-nl

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/test.en.tok.V, tests/test.fr.tok.V,
	tests/testpunctfilter.ok, tests/testpunctuation.ok: save current
	state. 1 problem left...

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx,
	tests/abbreviations.nl.tok.V, tests/bug0083.nl.tok.V,
	tests/ligaturen.nl.tok.V, tests/normalisation.nl.tok.V,
	tests/normalisationUTF16.nl.tok.V, tests/nu.nl.tok.V,
	tests/ocr.nl.tok.V, tests/test.nl.tok.V, tests/testencoding2.ok,
	tests/testfolia.ok, tests/testslash.ok: save current state of
	affairs. still not quite good

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: code layout

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 6c1ad9e94acad2452ee089a9fdd14bb4ba6822b0 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Aug 16 17:59:12 2016
	+0200

2016-08-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: test recursive application of rules

2016-08-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: since we have the config in uctodata, we better
	check that!

2016-08-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore: ignore more

2016-08-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: calling tokenizeWord() with a wrong parameter
	only accidentaly worked :{

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added heavy debug option

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: hmm adding -wB to diff somehow defeated tokenizing

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testpunctfilter.ok, tests/testquotes.ok,
	tests/testquotes2.ok: adapted results

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testpunctuation.ok, tests/testpunctuation.txt: added more
	tests

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* configure.ac: version bump after release

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: README update

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: README update

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* configure.ac: autoconf update to facilitate debian packaging

2016-07-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2016-07-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: bug fix announcement

2016-07-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: moved uctodata test after other tests, so that
	PKG_CONFIG_PATH is set correctly

2016-07-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2016-07-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac: NEW, we release v0.9

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: huh?

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: typo

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: fix travis

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: improved errormessage.

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am, src/ucto.cxx: per default, only install
	tokconfig-generic and its includes. Use 'generic' when no language
	is specified.

2016-06-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: interactive prompt is only shown when runnning
	from a TTY now input redirection still works as expected

2016-06-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, src/tokenize.cxx: add libreadline support when
	running from stdin

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-de, config/tokconfig-es, config/tokconfig-fr,
	config/tokconfig-fy, config/tokconfig-generic, config/tokconfig-it,
	config/tokconfig-nl-sonarchat, config/tokconfig-nl-twitter,
	config/tokconfig-nl-withplaceholder, config/tokconfig-pt,
	config/tokconfig-ru, config/tokconfig-sv, config/tokconfig-tr: added
	'magical-micorsoft-quotes' to all languages (not all tested!)

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: as said: added a rule ...

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/dimin.nl.tok.V, tests/dimin.nl.txt: added a rule for
	diminutives like CD'tje en baby'tjes. Also added tests

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/suffix.nl.tok.V, tests/suffix.nl.txt: added more suffix
	tests.

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: acute accents are used as quote's too :{

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: added an extra 'quote' to the rules
	concerning ' and the like

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: Don't check BOM on interactive input!

2016-06-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: added --help and --version

2016-06-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testencoding2, tests/testencoding2.ok: added an extra test.
	very weird and inconvenient Windows file

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: better handling of DOS files with CR

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testencoding2.ok: fixed UTF16BE (for the 1
	testcase we have...)

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: improved UTF16 support BE still not working

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/normalisationUTF16BE.nl.tok.V,
	tests/testencoding2, tests/testlanguage.ok: for now, we throw an
	encoding error on UTF16BE it is hard...

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/normalisationUTF16BE.nl.tok.V: added one more test, for BIG
	Endian. It FAILS!

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added comment

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: more refactoring

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: removed unused
	checkBOM() variant. some refactoring doen

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, tests/W1258.nl,
	tests/normalisationUTF16.nl.tok.V, tests/testencoding2,
	tests/testencoding2.ok: some code refactoring: check BOM only once.
	removed allatonce from tokenizeStream(( added poor-mans solution for
	UTF16. Should be tested on more files/platforms

2016-05-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small refactoring

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: started some
	refactoring

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: only check the BOM for files, and only for the
	first line

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testconf1.ok: code cleanup. Fixed minor
	change in test result.

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: more fancy file loolup. fallback to default
	configdir.

2016-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: the tests now survive changes in libfolia version
	numbers

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testall: removed debug line. Make sure new
	test is realy run.

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testclassnormalization, tests/testclassnormalization.ok: 
	added a test

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: added a
	--normalize option. This allows for every word in a certain
	TokenClass to be mapped to that class' name

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac: NEWS, again a new release

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore: ignore README

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-generic,
	config/tokconfig-it, config/tokconfig-nl-sonarchat,
	config/tokconfig-nl-twitter, config/tokconfig-nl-withplaceholder,
	config/tokconfig-pt, config/tokconfig-ru, config/tokconfig-sv,
	config/tokconfig-tr: added adapted ABBREVIATION rule to alls
	languages

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/{issue04.nl.txt => issue04.nl.txt.FAILS}: 
	added extra debug lines.  move issue04 out of the tests. It is known
	to fail AND WONTFIX

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue04.nl.tok.V, tests/issue04.nl.txt: added a new test

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue03.nl.tok.V, tests/issue03.nl.txt: added test for
	issue03 (fixed already)

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/bug0051.nl.tok.V, tests/bug0052.nl.tok.V,
	tests/bug0065.nl.tok.V, tests/bug0065b.nl.tok.V,
	tests/buggie01.nl.tok.V, tests/buggie01.nl.txt, tests/ocr.nl.tok.V,
	tests/quotetest_folgert.nl.tok.V,
	tests/quotetest_folgert2.nl.tok.V,
	tests/quotetest_folgert3.nl.tok.V,
	tests/quotetest_folgert4.nl.tok.V,
	tests/quotetest_folgert5.nl.tok.V,
	tests/quotetest_folgert6.nl.tok.V,
	tests/quotetest_folgert7.nl.tok.V,
	tests/quotetest_multiline.nl.tok.V,
	tests/quotetest_multisentence.nl.tok.V,
	tests/quotetest_nested2.nl.tok.V,
	tests/quotetest_onesentence.nl.tok.V, tests/test.de.tok.V,
	tests/test.en.tok.V, tests/test.es.tok.V, tests/test.fr.tok.V,
	tests/test.nl.tok.V: adaped to change in verbose output

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: modified ABBREVIATION rule. A ,: or ; may
	follow directly

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small fix in verbose output: no newline at
	ENDOFSENTENCE when NOSPACE too

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue01.nl.tok.V, tests/issue01.nl.txt: more usefulle name

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac: bumped version after release. added NEWS

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/buggie01.nl.tok.V, tests/nu.nl.tok.V: 
	merged bug fixes from 'generic'

2016-04-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-en, src/tokenize.cxx: reimplemented regexp stuff,
	but no improvement yet

2016-04-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/buggie01.nl.tok.V: added a "good" result

2016-04-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit b25b97c646df98715b93c6b2440de41d58130962 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Apr 19 16:27:57 2016
	+0200

2016-04-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/buggie01.nl.txt: added a new test

2016-04-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: addapted testresults to recent libfolia

2016-03-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed problem with passThru on an empty line

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: removed debug line, now the test is fixed

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: tru another testmethod

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: added -B option to diff

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: output 1 of the files that seem to fail on travis....

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: news!

2016-03-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 45c9ddfa67de37eefa775c0594153215351f255b Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Mar 8 16:54:51 2016
	+0100

2016-03-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: adapted tests to libfolia version change

2016-03-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: this branch needs libfolia 1.0

2016-03-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/folia1.xml, tests/folia2.xml,
	tests/folia3.xml, tests/slashes.xml: replaced all div-annotation by
	division-annotation

2016-02-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: use KWargs to instantiate folia elements

2016-02-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok, tests/testfolia2.ok: small diff in results
	because of newer folia version.

2016-02-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/Makefile.am, src/tokenize.cxx: Rule
	Of Three!

2016-02-19  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: oops, ucto had wrong lamabadge

2016-02-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 9f4405001213e0a31c05c677780d3e2fab8af4d9 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Thu Feb 18 16:49:26 2016
	+0100

2016-02-17  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 736d144b0a4d8c8da9626554a1dac9df53b443a3 Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Wed Feb 17 14:03:39 2016
	+0100

2016-02-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: oesp

2016-02-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit cbc8840d0d48d301e22d5b0c0638e9d90d720589 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Feb 3 12:08:13 2016
	+0100

2016-02-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, include/ucto/tokenize.h, src/tokenize.cxx,
	src/ucto.cxx, src/unicode.cxx: a 'generic' branch to wotrk with the
	new 'generic' libfolia

2016-02-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: test only the master

2016-02-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: test only master

2016-02-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: added more includes for the
	'generic' branches

2016-01-20  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-generic,
	config/tokconfig-it, config/tokconfig-pt, config/tokconfig-ru,
	config/tokconfig-sv, config/tokconfig-tr: added meta rules to all
	language configs

2016-01-20  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-it: aggiunto abbreviazioni italiani e regoli
	metà

2016-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testfolia.ok, tests/testfolia2.ok,
	tests/testfoliain.ok, tests/testslash.ok: adapted tests to newer
	libfolia version

2016-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: Bumped version after release

2016-01-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore, Makefile.am, NEWS, configure.ac: from SVN to Git

2016-01-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/test.nl.tok.V, tests/test.nl.txt: 
	changed rules added two more tests

2016-01-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: cleaner code

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/test.nl.tok.V, tests/test.nl.txt: added
	a new rule and test

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: we now support %include from different locations
	(real paths) lot of code reformatting. generally use {} in if
	statements.

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx, tests/test.nl.tok.V,
	tests/test.nl.txt: added another rule, and a test, of course

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/nl_afk.abr, config/tokconfig-nl, tests/test.nl.tok.V,
	tests/test.nl.txt: added some more abbrev's added a
	ABBREVIATION-COMPOUND rule and a testcase

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/nl_afk.abr, config/url.rule, tests/urls.nl.tok.V,
	tests/urls.nl.txt: added wnd to dutch abbreviations fixed url rule
	and added testcase

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added default: to switch

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: rules are stored in a
	map now.  Ehen multiple rules with the same name occur, the last one
	is kept.

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed logging

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: new style config file for dutch.  with
	meta-rules

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added a configurable split token to meta-rules
	default is % can be changed per rule

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: continued work on
	META-RULES

2016-01-08  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 14e0a858a5a02c7e5633d246e5c31730690d21d8 Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Fri Jan 8 09:40:07 2016
	+0100

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx: added code to parse an use
	META-RULES.  Works for 1 rule yet.  So don't use it!

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx: adapted 2 rules.
	introduces a dutchism in the code (good for at least English too, i
	think) More generic solution needed

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/url.rule: added a top level and cleaned up too

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/url.rule, tests/urls.nl.tok.V, tests/urls.nl.txt: fixed url
	rules.  added tests

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx, tests/suffix.nl.tok.V,
	tests/suffix.nl.txt, tests/urls.nl.tok.V, tests/urls.nl.txt: some
	more fixes. Also added more tests

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/encoding.nl.tok.V: WORD-COMPOUND may be
	all numbers too now.  Moved DATE detection in front of WORD-COMPOUND
	This triggered 1 change in 1 test.

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: WORD-COMPOUND may also start with numers now.
	(16-jarige, 12-en-een-half) Als moved WORD-COMPOUND to hit before
	NUMBER-STRING

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed NUMBER-ORDINAL rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V: adapted to strickter rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: strickter rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: typo

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/ocr.nl.tok.V, tests/test.nl.tok.V: all
	tests green

2016-01-06  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 637ccc620f231e928c58f8c668c38e50e4579501 Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Wed Jan 6 13:41:25 2016
	+0100

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V: updated testresult

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit dd7e11259e76ea9f38837e9125c7e85581688f7a Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Jan 6 13:32:05 2016
	+0100

2016-01-06  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-nl: always keep hyphens in words (hyphen is not a
	separator)

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: added a NUMBER-COMPOUND rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/test.nl.tok.V, tests/test.nl.txt: added
	more tests (some failing yet)

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: added more word-compound
	tests

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: added -Z to the diff command.  Don't care about
	trailing whitespace

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: added -b to diff command. otherwise trailing whit

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt, tests/test.py: added a
	(failing!) test

2016-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore: added

2016-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* README, configure.ac, fink/ucto.info, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: updated copyright/disclaimer

2015-11-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: fixed includes

2015-11-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: handle leading spaces in passthru mode correctly

2015-11-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/passthru2.txt: added a leading space to test that it is
	skipped

2015-11-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* README: Small edits

2015-11-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit d6c2fb1388ccf36462e8c7c0d92bf7fd4ca9f238 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Nov 24 12:13:55 2015
	+0100

2015-11-24  Maarten van Gompel <proycon@anaproy.nl>

	* README: README update

2015-11-24  Maarten van Gompel <proycon@anaproy.nl>

	* README: README update

2015-06-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok: adapted tests to libfolia
	version 0.12 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18477
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml, src/tokenize.cxx: attempted fix for now non-existant
	FoLiAElement::deeptext().. will a normal text() do? (it's supposed
	to be deep by default now)? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18474
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl-withplaceholder: Added
	variant of dutch tokenizer that includes placeholders (variables in
	the form %identifier% that will be preserved as-is, useful for
	subtitutions after tokenisation) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18443
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml: tests
	have changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18421
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok: tests have changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18420
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: smal fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18416
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* README: updated instruction after flogging git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18414
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-18  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* README: added installation instructions git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18410
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-04-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: updated, modernized git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18335
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: adapted usage() git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18301
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: safeguard git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18300
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain, tests/testfoliain.ok: adapted test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18299
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to fix punctuation filter for FoLiA
	input git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18298
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain, tests/testfoliain.ok: adapted test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18297
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added experimental code to allow -l and -u on
	FoLiA input.  needs more testing!!!! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18296
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am: fall back to c++0x git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18266
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: small improvement git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18135
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18069
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: modernized. using
	C++11 features now! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18056
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am: C++11 aware git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18035
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: removed useless const.  typos in comments
	corrected.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18019
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: implemented smarter
	tokenizeSentenceStream() function.  Needed to fix flushSentences()
	to get it to work! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18017
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: added another
	tokenize() function. (reading tokenized strings froma steram) We
	realy need to weed these out! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18014
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bump after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18012
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: get ready for release 0.8 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18009
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 2015 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18006
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: allatonce enabled by default for tokenize() to
	folia doc git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17995
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h: 2015 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17993
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: added new
	tokenize(string,string) meta-function for the API git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17959
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: added FRACNUMBER too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17958
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-generic,
	config/tokconfig-it, config/tokconfig-nl,
	config/tokconfig-nl-twitter, config/tokconfig-pt,
	config/tokconfig-ru, config/tokconfig-sv, config/tokconfig-tr: 
	fixing date rules and adding FRACNUMBER git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17957
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: <Flo> zou je ook nog een variant
	dd/mm toe kunnen voegen? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17956
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: handle dates with slashes in twitter:
	< proycon> doen wij datums niet met hyphens officieel in het
	Nederlads?     <Flo>| heeft twitter maling aan ;) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17955
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-13  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-ru: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17934
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-13  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-ru: added Russian git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17933
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: fixed hyphens according to
	/usr/share/doc/groff-base/README.Debian git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17908
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: fixed hyphens according to
	/usr/share/doc/groff-base/README.Debian git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17907
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed terrible bug, also some code-cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17873
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17872
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/tokenize.cxx, tests/testfolia.ok,
	tests/testfolia2.ok: made ucto Part aware.  2 tests fail atm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17871
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, tests/partest.nl.tok,
	tests/partest.nl.txt, tests/partest2.nl.tok, tests/partest2.nl.txt,
	tests/partest2_folia.nl.txt, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.txt, tests/partest_folia.nl.xml,
	tests/test.py: Re-added memory saving code: tokens of entire input
	not retained in memory prior to output   +    added extra tests  +
	updated test.py framework to deal with FoLiA tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17865
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: yet another fix for travis git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17857
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: (travis-ci) show some logs after failure git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17856
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17854
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testone: tests return proper return codes now git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17852
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: travis-ci: forgot libfolia as dependency git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17851
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: fix for travis git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17850
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: travis update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17846
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: update for travis, output tests to stderr instead of
	stdout git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17845
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.tok.V, tests/test.de.txt: fixed german test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17844
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: another update for travis-ci git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17843
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: fix for travis-ci build git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17842
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: added travis-ci git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17841
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: shut up! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17840
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: added
	experimental punctuation filter.  Filters most punctuation. but NOT
	when rules like ABBREVIATION SUFFIX etc are applicable.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17839
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testpunctfilter, tests/testpunctfilter.ok: 
	added a test for the experimantal punctuation filter git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17838
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{quotetest_folgert8.nl.txt =>
	quotetest_folgert8.nl.txt.disabled}: disable this one too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17837
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{munten.nl.txt => munten.nl.txt.disabled},
	tests/{units.nl.txt => units.nl.txt.disabled}: disable 2 tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17836
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: Reverted the pevious 2
	patches.  The fist was wrong. The second didn't fix it.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17835
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testpunctuation.ok: accept suboptimal solution for now git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17834
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: more libfolia bumps git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17833
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok: bumped libfolia version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17832
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: attempted fix for FoLiA
	output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17756
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: Reducing the
	unnecessarily high memory usage, especially when no FoLiA output is
	requested git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17755
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: macro fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17746
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-19  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: added compiler fallback git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17737
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac, src/tokenize.cxx: bumping git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17695
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17687
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: clearer option handling, better checks, and fixed
	non-working -S option (nobody seems to use it) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17686
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: no more nasty segfault for ucto -h git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17684
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: small edit to satisfy clang git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17682
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17674
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: removed ugly hack git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17662
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: added a useful function to the API git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17654
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: removed declared, but undefined function git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17651
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-13  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: little alternative
	function working with pointers, needed for python-frog git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17648
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/ucto.cxx: use ticcutils 0.6 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17549
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-23  antalb <antalb@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/nl_afk.abr: added RK git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17546
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: switched to TiCC CommandLine stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17544
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-14  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: adapt to newer libfolia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17513
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: generally switched to
	UChar32 and Unicode codepoints.  removed some dead code too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17484
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added EMOTICON recognision. Needed some rework
	vor 2 codepoint long characters git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17483
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/smileys.nl.tok.V, tests/smileys.nl.txt, tests/testfolia.ok,
	tests/testfolia2.ok: added Emoticons to the smiley test.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17482
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-07-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some code moved around git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17453
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-07-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V, tests/testfolia.ok, tests/testfolia2.ok: some
	tests are changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17451
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-07-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: handle Note like Caption git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17418
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: adapted to new libfolia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17393
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-25  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/pt.abr, config/tokconfig-pt: Adicionei
	regras para tokenização portuguesa.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17387
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: avoid crash on empty inputfile git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17357
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tst.sh: hmm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17335
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac, include/ucto/tokenize.h, src/tst.sh: NEWS
	fixed 'make distcheck' git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17332
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: and reinstated git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17245
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: removed soft link git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17244
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: made getSentence() public git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17237
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: moved folia include to header git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17236
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: removed "config.h" git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17235
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, include/ucto/tokenize.h, src/tokenize.cxx: added to
	facilitate secret project ;) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17234
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: add git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17181
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17180
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/Makefile.am, src/tst.sh: fixed 'make check' (but
	only for automake 1.13 and up) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17177
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, Makefile.am: fixed 'make dist' git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17175
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-01-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/ligatures.filter, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: Happy Newyear! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16823
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-12-16  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-tr: added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16771
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/unicode.cxx: compiler satisfaction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16652
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* bootstrap => bootstrap.sh, configure.ac: modernized git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16634
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.es.tok.V, tests/test.fr.tok.V: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16574
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-fr, config/tokconfig-it,
	src/tokenize.cxx: fixes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16573
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-it, config/tokconfig-nl,
	config/tokconfig-nl-twitter, config/tokconfig-sv, src/tokenize.cxx,
	tests/test.pt.txt.disabled: Case insensitivity does not work, libicu
	too buggy, made rules explicitly case insensitive git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16572
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Do NOT replace existing <t> content.  Added
	Caption to exceptions for addign <p> nodes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16564
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: test has changed after
	decision about embedden stuff in <t> nodes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16563
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed ListItem stuff.  Also DON'T add a
	paragraph to Head elements git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16562
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: keep track of line
	numbers in verbose output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16560
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-generic: generic update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16525
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es, tests/test.es.tok.V,
	tests/{test.es.txt.disabled => test.es.txt}, tests/test.fr.tok.V: 
	spanish update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16524
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16523
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr, tests/{test.fr.txt.disabled => test.fr.txt}: 
	changes in french tokenisation (suffixes) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16522
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V: oops git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16521
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V: test result git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16520
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-fr,
	tests/test.en.txt: updating configurations git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16519
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/tokenize.cxx: readded (?i) , test still to be
	added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16518
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-08-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: added a reset() member.
	Enables a fresh start.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16418
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-08-01  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-generic: added generic
	language-independent tokeniser configuration, for unsupported
	languages git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16407
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-08  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: hmpfff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16331
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-08  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: modified tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16330
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: small change in testresult git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16307
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed problem with empty text git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16304
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/empty.xml, tests/testfoliain, tests/testfoliain.ok: added a
	test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16303
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: re-re-refixed folia stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16297
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia6.xml, tests/testfoliain, tests/testfoliain.ok,
	tests/testslash.ok: modified and extended test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16296
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: oesp git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16295
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: adapted test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16294
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/ucto.cxx: sorted out textclass stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16293
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/ocr.nl.tok.V, tests/ocr.nl.txt: added a test with some
	terrible stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16292
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: changed tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16291
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: folia handling improved: - separate --inputclass and --outputclass options - better <t> text added on <p> and  <s> git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16290
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain, tests/testfoliain.ok: added a test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16281
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed outputclass stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16280
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testoption1.ok, tests/testslash.ok, tests/testusage.ok: 
	modified tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16279
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: replaced the --textclass opion.
	Use --inputclass ans --outputclass! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16278
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testpunctuation, tests/testpunctuation.ok,
	tests/testpunctuation.txt: added another test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16231
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: keep fiddling with EOS stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16230
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: desired result is changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16226
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/testencoding2.ok: hacking the eosmarkers
	again git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16225
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/standard-eos.eos, include/ucto/tokenize.h,
	src/tokenize.cxx: added some ugly hacks to improve EOS detection
	when quotes are in sight.  A code cleanup is very desirable! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16224
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/qtest.nl, tests/testall, tests/testquotes.ok,
	tests/testquotes2, tests/testquotes2.ok: added more tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16223
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: removed comment git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16201
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V: modified test result git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16186
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: solve the Alex' and Frans' problem with an
	extra RULE (only match words with an Uppercase) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16185
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V, tests/suffix.nl.txt: modified test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16175
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V: hmm..  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16173
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: added more SUFFIX-es git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16172
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V, tests/suffix.nl.txt: new test added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16171
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to make more silly-quote proof git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16168
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/unicode.h, src/tokenize.cxx, src/unicode.cxx,
	tests/encoding.nl.tok.V: updated unicodefilter API git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16163
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-05-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fix ENDOFSENTENCE problems in combination with
	punctuation.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16140
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-05-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/qtest.nl, tests/testfolia.ok, tests/testfolia2.ok,
	tests/testfoliain.ok, tests/testquotes.ok, tests/testslash.ok: 
	adaptes tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16139
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bump version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15945
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: wrong date in NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15943
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am: increment current library version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15925
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.nl.tok.V: adapt to new tokenisation rules git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15918
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: changed folia include path git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15910
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15905
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: we need the most recent ticcutile git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15885
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: modified folia include path git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15875
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* : tags git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15851
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: more quote hacking git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15809
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testquotes.ok, tests/testslash.ok: some rule neams have
	changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15807
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: small edits git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15806
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx, tests/testquotes.ok,
	tests/testslash.ok: some rexexp patches for words like A50,
	vanessa23 etc.  adaptes tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15805
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: more quotes for '40 etc.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15803
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: small fix for prime's/quotes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15802
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added small hack to appen 'dangling' quotes to
	the previous sentence.  This avoids sentences with only a ' or a ".
	(extend to exotic quotest too?) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15797
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: be sure to take "current" as the default class for
	text searching git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15774
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: oesp.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15773
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-it: fixed ABBREVIATION rule
	for other languages git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15769
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: Updated.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15766
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: added a
	--textclass option git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15765
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia5.xml, tests/testfoliain, tests/testfoliain.ok,
	tests/testoption1.ok, tests/testusage.ok: added test for the new
	--textclass option git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15764
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added the passthru fix for FoLia docs also
	elsewhere git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15693
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: passtru mode was handled incorrectly for FoLiA
	input git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15683
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: make sure that we tokenize existing FoLiA in our
	own set! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15682
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-14  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am: added to makefile git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15675
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-14  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/es.abr, config/tokconfig-de, config/tokconfig-en,
	config/tokconfig-es, config/tokconfig-fr: updated foreign
	tokenisation configurations git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15674
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some debugging lines silenced git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15648
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bracket.nl.tok.V, tests/bracket.nl.txt: added a test for the
	EOS in brackets problem.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15646
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/test.py: added some trickery to transfer
	EOS markers to next token if it is a closing bracket. EXPERIMENTAL git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15645
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testone: improved test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15587
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/utf8bom.nl: we need this files too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15586
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: now we handle (and use)
	BOM markers.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15585
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testencoding2, tests/testencoding2.ok: added a test for the
	BOM git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15584
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h,
	src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: Bump year git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15571
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-06  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: wrong man git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15532
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/nl_afk.abr: removed 'al' beacuse it isn't an abrivation,
	ans it IS a word! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15529
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/testfolia.ok, tests/testfolia2.ok,
	tests/testfoliain.ok, tests/testslash.ok: use datetime='now()' in
	declarations. And test that too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15528
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: add datetime attribute to token-annotation
	declaration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15522
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: only add a set default definition when there is
	something to do git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15520
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: removed odd typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15519
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/{nl_afk => nl_afk.abr}: fixed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15518
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am: added to config data git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15490
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/nl_afk, config/tokconfig-nl-twitter: adding forgotten file git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15489
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-sonarchat, config/tokconfig-nl-twitter: use
	the nl_afk abbreviations file git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15480
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, include/ucto/tokenize.h, src/tokenize.cxx: 
	implemented %include for ABBREVIATIONS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15479
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: docupdate git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15478
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: updated after comments git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15472
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.bib, docs/ucto_manual.tex: Added documentation
	(finally) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15468
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/reverse-smiley.nl.tok.V, tests/reverse-smiley.nl.txt,
	tests/smileys.nl.tok.V, tests/smileys.nl.txt: added a test for
	reversed smileys.  addapted the smiley testbtoo.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15397
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/smiley.rule: attempt to support 'repetitive' smileys like
	this :))))) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15396
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-sonarchat, config/tokconfig-nl-twitter: added
	REVERSE-SMILEY rule git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15395
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/smiley.rule, config/tokconfig-nl: attempt to define a rule
	for reverse smileys {-8.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15394
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: get in line with standard
	tokconfig-nl git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15387
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testtwitter, tests/testtwitter.ok,
	tests/twitter.txt: added a test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15386
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* bootstrap: bootstrap fix (matched failed on automake 1.11.6 with
	Ubuntu 12.10) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15361
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: adapted to changed FoLiA stylesheet handling git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15355
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: adapted to improved libfolia output (explicit
	UTF-8 encoding) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15278
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h: safeguarded the
	safeguards git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15275
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-09-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: unneeded check git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15158
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-08  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, configure.ac, include/ucto/tokenize.h, src/tokenize.cxx: 
	use ticcutils in TiCC namespace git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15062
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{test.es.txt => test.es.txt.disabled}, tests/{test.fr.txt
	=> test.fr.txt.disabled}, tests/{test.pt.txt =>
	test.pt.txt.disabled}: teporary disabled until Proycon does his job git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15027
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-en: improved english tokenization rules git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15026
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V: accaptable result git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15025
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de: V.S. en d.h. gaan nu goed. a.d.Donau
	misschien niet git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15024
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.tok.V: ik denk dat dit dichter tegen de waarheid
	zit.  Nu nog zorgen dat het er uit komt :{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15023
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.tok.V: new git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15022
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, src/tokenize.cxx: attempt to fix .de git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15021
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: keep it private git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15020
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: save intermediate
	refactored state git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15019
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15016
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: fixed eos problems.
	'-s' option seems to work now.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15015
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall: test the new test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15014
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/eos.txt, tests/testoption-s, tests/testoption-s.ok,
	tests/utt.txt: added a new test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15013
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: better info.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15012
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: typo, and some refactoring.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15011
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: small cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15010
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: further refactoring git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15009
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: re-re-re-factored :P git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15005
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: one small step… git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15004
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: more cleanup.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15003
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: some more refactoring,
	and added some comment git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15001
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: ok, more next week git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15000
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: next refactor step git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14999
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: hmm. progress… git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14998
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: save intermediate result during cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14997
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: refactoring: removed
	parameters without any visable changes in behaviour.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14996
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quotetest_folgert8.nl.tok.V: Correct the test case.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14995
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* AUTHORS: Honour Folgert git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14994
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: indentation and such git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14993
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: some renaming for my
	own mental health git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14992
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: remove unused functionm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14991
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: started some
	refactoring git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14990
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia2.txt, tests/testall, tests/testfolia2,
	tests/testfolia2.ok: added a complex folia test. asume the result is
	OK for now.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14989
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: I think this more in line with wat we
	want.  still doubtfull quotation git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14988
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to fix bug94 (fixes quotetest_folgert7
	too??) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14987
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quotetest_folgert7.nl.tok.V: I think the correct file wasn't
	correct at all. Folgert?? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14986
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed debugging lines git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14972
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14962
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed folia id's git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14961
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-03  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fy: partial revert git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14947
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-03  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fy: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14946
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-03  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fy: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14945
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-13  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testquotes: properties git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14861
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-13  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0094.nl.tok.V, tests/bug0094.nl.txt: new bug found git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14860
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0051.nl.tok.V, tests/bug0052.nl.tok.V,
	tests/bug0065b.nl.tok.V, tests/quotetest_folgert.nl.tok.V,
	tests/quotetest_folgert2.nl.tok.V,
	tests/quotetest_folgert3.nl.tok.V,
	tests/quotetest_folgert4.nl.tok.V,
	tests/quotetest_folgert5.nl.tok.V,
	tests/quotetest_folgert6.nl.tok.V,
	tests/quotetest_folgert7.nl.tok.V,
	tests/quotetest_folgert8.nl.tok.V,
	tests/quotetest_multiline.nl.tok.V,
	tests/quotetest_multisentence.nl.tok.V,
	tests/quotetest_nested2.nl.tok.V,
	tests/quotetest_onesentence.nl.tok.V, tests/test.nl.tok.V,
	tests/testquotes.ok: addapted to changed verbose output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14847
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: ok, we are more happy now, but there is
	still a BEGINOFSENTENCE missing I think git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14846
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed indentation of the source also always put
	an extra newline after ENDOFSENNTENCE in verbose mode git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14845
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/nu.nl.tok.V: dit is al meer wat we willen git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14844
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.py: reversed parameters to diff.  now we do 'diff <what
	we got> <what we want>" This is more in line with other tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14843
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-04  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, config/tokconfig-nl, src/tokenize.cxx,
	tests/quotetest_folgert5.nl.tok.V, tests/quotetest_folgert5.nl.txt,
	tests/quotetest_folgert6.nl.tok.V, tests/quotetest_folgert6.nl.txt,
	tests/quotetest_folgert7.nl.tok.V, tests/quotetest_folgert7.nl.txt,
	tests/quotetest_folgert8.nl.tok.V, tests/quotetest_folgert8.nl.txt: 
	Fix nested quoted sentences; added testcases; small additions to
	tokconfig-nl git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14833
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-31  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: update header, u_isquote git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14826
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-31  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: getting used to svn...  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14825
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-31  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quotetest_folgert4.nl.tok.V,
	tests/quotetest_folgert4.nl.txt: small fix in quote detection,
	testcase added.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14824
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/Makefile.am: exterminated AC_DEFINE_DIR here too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14823
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14820
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/bug0065b.nl.tok.V,
	tests/quotetest_folgert.nl.tok.V, tests/quotetest_folgert.nl.txt,
	tests/quotetest_folgert2.nl.tok.V, tests/quotetest_folgert2.nl.txt,
	tests/quotetest_folgert3.nl.tok.V, tests/quotetest_folgert3.nl.txt,
	tests/test.py: Folgert fixed quote detection, extra tests added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14819
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: handle ' just like " in quote-detection git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14777
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/standard-quotes.quote: updated comment git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14776
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testquotes.ok: let's call this OK, for now.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14775
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/qtest.nl, tests/qtest.spec, tests/testall, tests/testquotes: 
	added tests for quotes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14774
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: attempt to fix 'time' problem git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14721
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/smiley.rule, tests/smileys.nl.tok.V: fixed smiley git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14720
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/datetime.nl.tok.V, tests/datetime.nl.txt,
	tests/smileys.nl.tok.V, tests/smileys.nl.txt, tests/testall,
	tests/testconf2.ok: added tests (still fail!) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14719
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-04-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/slashes.xml, tests/testfolia.ok, tests/testfoliain.ok,
	tests/testslash.ok: adapted to new folia version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14703
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bump version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14587
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: get ready for release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14585
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: numb change git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14472
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfoliain.ok, tests/testslash.ok: 
	newer libfolia version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14471
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-fy: added frisian
	tokenisation (untested yet) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14432
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14356
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: News! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14342
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: bit of cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14336
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: formatting git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14322
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/Makefile.am, tests/slashes.xml, tests/testall,
	tests/testfoliain, tests/testslash, tests/testslash.ok: added a test
	for wierd \\\ problems git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14304
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx: replaced all ?i stuff in
	regular expressions.  in an attempt to avoid ICU bug git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14302
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl-sonarchat: added for sonar
	chat tokenisation git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14292
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: maarten mangled the test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14287
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml, tests/folia2.xml, tests/folia3.xml,
	tests/folia4.xml, tests/testfoliain, tests/testfoliain.ok: test
	update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14286
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex, include/ucto/tokenize.h, src/tokenize.cxx: 
	folia input fixes (regarding event tag) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14285
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfoliain.ok: adapted to folia 0.8 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14284
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-08  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14223
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-08  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: added beginnings of ucto manual git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14222
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/ucto.cxx, tests/testoption1.ok, tests/testusage.ok: 
	added a -X and an --id= option. this is more in line with frog.  -x
	<id> still works git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14159
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx: added 'escape' possibility
	to config files now we can read: [ATTACHEDSUFFIXES] \['`’]s
	\['`’]t \['`’]n which means: [ATTACHEDSUFFIXES] ['`’]s ['`’]t ['`’]n git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14145
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml, tests/folia2.xml, tests/folia3.xml,
	tests/testfolia.ok, tests/testfoliain.ok: fixed tests. libfolia is
	strict on <div> nodes (because FoliA.pdf demands it) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14144
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr, config/tokconfig-nl,
	config/tokconfig-nl-twitter: support for unicode apostrope 0x2019 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14134
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: Make debugging output
	go to theErrlog generally git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14109
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* ucto.spec: was moved to Timbl-dev/rpm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13991
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: cleanup configure mess git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13944
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after Release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13934
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, config/Makefile.am, config/smiley.rule,
	config/tokconfig-nl, config/tokconfig-nl-twitter: put SMILEY in a
	separate .rule file.  It is language independent.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13918
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: added a much more powerfull SMILEY rule.
	Which also works with older ICU cversions! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13917
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Even better error message git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13916
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf2.ok, tests/testfolia.ok, tests/testfoliain.ok: 
	adapted to small changes in output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13915
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: slighly better error message git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13914
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h,
	src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: next year git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13842
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13762
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: bumped version somehow 0.4.8 was already
	released git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13756
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-14  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: be more informative git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13698
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: libfolia API is a moving target git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13652
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, include/ucto/tokenize.h, src/tokenize.cxx: adapted
	to major change in libfolia: AbstractElement ==> FoliaElement git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13648
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfoliain.ok: adapted to new folia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13647
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: test results changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13613
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml: stray " git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13612
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml, tests/folia2.xml, tests/folia3.xml: replaced \n
	wilt real newline character &#xA; git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13611
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: attempt to fix -F. still uncertain git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13610
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{folia.xml => folia1.xml}, tests/folia2.xml,
	tests/folia3.xml, tests/testfoliain, tests/testfoliain.ok: added and
	changed -F tests Still not sure if this is wahat we want git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13609
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl-twitter: Added specicial
	tokeniser configuration for Twitter (dutch) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13608
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, config/url.rule, tests/smileys.nl.tok.V,
	tests/smileys.nl.txt, tests/urls.nl.tok.V, tests/urls.nl.txt: Added
	URL-DOMAIN and SMILEY detection (enabled only in tokconfig-nl for
	now) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13607
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia.xml, tests/testall, tests/testfoliain,
	tests/testfoliain.ok: added simple test for the _F FoLiA XML input
	option git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13605
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1, src/ucto.cxx: small change in usag() Enhanced man
	page git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13601
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testoption1.ok, tests/testusage.ok: again output was changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13600
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13599
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13598
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testoption1.ok, tests/testusage.ok: output has changed, so
	adapt test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13597
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: removed 2 compiler warnings 2 others still
	there: Proycon, what were you thinking? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13596
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: completed FoLiA input
	support (needs more testing though) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13587
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: tokenisation of sentences in existing FoLiA now
	works git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13586
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-25  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixes for FoLiA input support git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13585
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-25  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: 
	Attempting to add FoLiA read support to ucto.. not entirely finished
	and tested yet! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13583
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added some hacks to avoid a nasty ICU bug.  see
	http://bugs.icu-project.org/trac/ticket/8824 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13536
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13511
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am, src/tst.sh: more lost stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13510
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* Makefile.am, tests/Makefile.am, tests/bug0083.nl.tok.V,
	tests/bug0083.nl.txt, tests/test.de.txt, tests/test.en.txt,
	tests/test.es.txt, tests/test.fr.txt, tests/test.pt.txt,
	tests/tst.cfg, tests/tst.ok, tests/tst.txt: hmm lost a lot of stuff
	in the merge :{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13509
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* Makefile.am, NEWS, config/Makefile.am, config/tokconfig-en,
	config/tokconfig-es, config/tokconfig-fr, config/tokconfig-generic,
	configure.ac, docs/ucto.1, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/Makefile.am, src/tokenize.cxx,
	src/tst.sh, src/ucto.cxx, src/unicode.cxx, tests/Makefile.am,
	tests/bug0083.nl.tok.V, tests/bug0083.nl.txt,
	tests/quoteproblem.txt, tests/test.de.txt, tests/test.en.txt,
	tests/test.es.txt, tests/test.fr.txt, tests/test.pt.txt,
	tests/test.py, tests/testencoding2, tests/testfolia,
	tests/testfolia.ok, tests/testinclude, tests/testnormalisation,
	tests/testone, tests/testoption1.ok, tests/testoption2,
	tests/testpassthru, tests/testusage.ok, tests/tst.cfg,
	tests/tst.ok, tests/tst.txt, ucto.pc.in: Merged the folia branch
	into ucto.  ucto now always requires libfolia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13508
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some layout changed and slack removed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13462
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx: small changes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13459
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: made compiler happy.  Removed dead code git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13449
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es: support for spanish ¿ and ¡ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13366
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13365
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13349
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-generic: generic tokeniser configuration, not for
	any particular language  (not recommended) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13348
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-en: English tokenisation: wo n't  , do n't git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13347
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es: Added spanish configuration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13346
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr: Added french abbreviations git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13345
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr: updates to french configuration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13344
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-en, tests/test.en.txt: update English
	configuration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13342
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.txt: correction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13341
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.txt: correction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13340
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.txt: generic german test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13339
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.pt.txt: correction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13336
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.pt.txt: corrections git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13335
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.pt.txt: Generic portuguese test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13334
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.es.txt: Generic Spanish test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13333
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.fr.txt: more french git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13332
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.fr.txt: Initial generic french test.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13331
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.txt: generic English test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13330
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/Makefile.am: last small oesp git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13189
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* Makefile.am, configure.ac, src/Makefile.am, src/tst.sh,
	tests/Makefile.am, tests/tst.cfg, tests/tst.ok, tests/tst.txt: 
	attempt to provide 'make check' and 'make distcheck' git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13188
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after Release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13043
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: attempt for more precise desired version\ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12956
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V, tests/bug0083.nl.txt: added an other
	quoteing problem and the disered output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12955
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12950
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: release! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12949
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: the API should have the same defaults ofcourse git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12942
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.py, tests/testfolia, tests/testoption1.ok,
	tests/testusage.ok: adapted test to changed QuoteDetection regime.
	hardcoded -Q in test.py, arghh.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12940
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, docs/ucto.1, src/tokenize.cxx, src/ucto.cxx: Reversed
	working of -Q Updated man more NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12938
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: applied
	same changes to readsettings an init as in the libfolia branch git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12931
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf1.ok, tests/testconf2.ok, tests/testfiles1.ok,
	tests/testinclude.ok, tests/testlanguage.ok: outpu is changed a
	little bit git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12930
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-18  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx, tests/quoteproblem.txt,
	tests/quotetest_nested2.nl.tok.V, tests/quotetest_nested2.nl.txt: 
	Fixed nested quote issue (unbalanced quote/sentence issue) and build
	in extra protections. Incl. test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12909
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-07-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/abbreviations.nl.tok.V, tests/bug0048.nl.tok.V,
	tests/bug0051.nl.tok.V, tests/bug0052.nl.tok.V,
	tests/bug0054.nl.tok.V, tests/bug0063.nl.tok.V,
	tests/bug0065.nl.tok.V, tests/bug0065b.nl.tok.V,
	tests/bug0065c.nl.tok.V, tests/bug0067.nl.tok.V,
	tests/bug0075.nl.tok.V, tests/datetime.nl.tok.V,
	tests/encoding.nl.tok.V, tests/genitive.nl.tok.V,
	tests/ligaturen.nl.tok.V, tests/multisentenceoneline.nl.tok.V,
	tests/normalisation.nl.tok.V, tests/nu.nl.tok.V,
	tests/onesentencemultiline.nl.tok.V, tests/parentheses.nl.tok.V,
	tests/quotetest_multiline.nl.tok.V,
	tests/quotetest_multisentence.nl.tok.V,
	tests/quotetest_nested.nl.tok.V,
	tests/quotetest_nosentence.nl.tok.V,
	tests/quotetest_onesentence.nl.tok.V, tests/test.nl.tok.V,
	tests/urls.nl.tok.V: fixed ucto tests \o/ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12749
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-06-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quoteproblem.txt: added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10447
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10238
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testfolia.ok, tests/testone,
	tests/testoption1.ok, tests/testpassthru, tests/testpassthru.ok,
	tests/testusage.ok: added and improved tests for passthru updated a
	few .ok files.  I hope proycon fixes all .V.tok files git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10237
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: fixed encoding problem with
	passthru mode.  Added --passthru option to usage() git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10236
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to handle Windows™ files with \r\n.
	Just strip the last \r git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10223
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest_crlf.nl.txt: converted the file to CRLF and set the
	svn:eol-style property to CRLF git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10220
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest_crlf.nl.tok.V, tests/partest_crlf.nl.txt: added test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10214
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: I think I fixed bug77 (the NOSPACE bug), but now
	all tests fail (understandably), they will have to be checked
	thoroughly checked and updated.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10209
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia.txt, tests/testall, tests/testfolia,
	tests/testfolia.ok: added a simple FoLiA test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10156
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx: Fix: proper XML entities in FoLiA output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10121
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10015
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: set date for release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10013
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9980
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: avoid crash when no new tokens are found because
	of emptyp words (this might happen on Mac and DOS files with \r line
	separators ) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9979
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: ok, this is the ugly code :{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9871
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testpassthru.ok: sort of fixed <utt> handling in passthru
	code gets ugglier every day ;{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9867
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/passthru2.txt, tests/testpassthru, tests/testpassthru.ok: 
	added a better passthru test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9866
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: more News git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9863
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: small change in configfile
	handling.  Now the messages to stdout are correct.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9862
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf1.ok, tests/testconf2.ok, tests/testinclude.ok: 
	results ar slighlty different after small code reshuffle git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9861
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/invoer.rule, tests/small.cfg, tests/small.txt,
	tests/testall, tests/testinclude, tests/testinclude.ok: added a test
	for inclusion git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9855
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf1.ok: small change after bug fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9854
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed include file problem. I think. :) When a
	configfile with a (absolute/relative) path is provided (-c option)
	the include files will be searched in the same dir as that config
	file git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9853
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{bug0072.nl.tok.V => bug0075.nl.tok.V},
	tests/{bug0072.nl.txt => bug0075.nl.txt}: wrong name for this bug git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9845
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: KNOWN Abbreviations are case sensitive now.
	Fixes bug 72 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9844
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0072.nl.tok.V, tests/bug0072.nl.txt: added bug72 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9843
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-27  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: version bump after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9732
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-27  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9730
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-27  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Implemented sentenceperlineinput support
	directly in DetectSentenceBounds and PassTru (previously
	unsupported) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9729
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: also %include for
	[EOSMARKERS] git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9591
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/exotic-eos.eos,
	config/exotic-quotes.quote, config/ligatures.filter,
	config/standard-eos.eos, config/standard-quotes.quote,
	config/tokconfig-de, config/tokconfig-nl: added common files for
	inclusion in config git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9590
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl, include/ucto/tokenize.h,
	src/tokenize.cxx: now [FILTER] and [QUOTES] also honour %include
	requests in the config For [FILTER] the file extension MUST be
	.filter For [QUOTES] the file extension MUST be .quote git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9589
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx: fixed bug 67.  Now it is
	possible to add an empty replacement in the [FILTER} block git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9582
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0067.nl.tok.V, tests/bug0067.nl.txt: added a new bug test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9581
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: no schemalocation for FoLiA git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9350
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: arghh git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9313
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: hmpff.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9311
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, include/ucto/tokenize.h, src/ucto.cxx: added functions to
	display version info.  Updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9310
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: return conforming exit status.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9309
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9271
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: news! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9270
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/bug0065.nl.tok.V,
	tests/bug0065b.nl.tok.V, tests/bug0065b.nl.txt,
	tests/bug0065c.nl.tok.V, tests/bug0065c.nl.txt: fixed bug0065 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9268
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0065.nl.txt: new bug git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9267
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx, src/unicode.cxx: Ok, now ALL unicode
	strings are output as UTF8.  Just to be sure.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9261
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Attempt to fix output encoding problems. We want
	it to be UTF8.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9258
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* moved ucto to its final destination git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9240
	12f355fe-0486-481a-ad91-c297ab22b4e3

