diff options
author | Ludovic Pouzenc <ludovic@pouzenc.fr> | 2014-09-20 09:17:18 +0200 |
---|---|---|
committer | Ludovic Pouzenc <ludovic@pouzenc.fr> | 2015-04-14 07:44:29 +0200 |
commit | d6f22a2af48f83d63b5381118d2029797458194e (patch) | |
tree | cb6bef9a98335a7af2aee40b0752d14fcee0916e | |
parent | 774194091e9bcee08e48fcdf4127f9afd9d6d644 (diff) | |
download | sssync-d6f22a2af48f83d63b5381118d2029797458194e.tar.gz sssync-d6f22a2af48f83d63b5381118d2029797458194e.tar.bz2 sssync-d6f22a2af48f83d63b5381118d2029797458194e.zip |
Early development stages (before SCM) : WIP_1
Early development stages (before SCM) : WIP_2
Early development stages (before SCM) : WIP_3
Early development stages (before SCM) : WIP_4
Early development stages (before SCM) : WIP_6
Early development stages (before SCM) : WIP_7
Early development stages (before SCM) : WIP_8
Adds documentation folder as an Eclipse project.
Adds README for github.
Decent source tree by tuning Eclise project's location
One forgetten file while movign everything :)
Adding Copyright, licencing (GPL v3), correcting README
89 files changed, 5921 insertions, 1 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d043e7e --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +*~ +.metadata/ +src/*/bin/ +src/*/.pmd +build/* @@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/> + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + <program> Copyright (C) <year> <name of author> + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +<http://www.gnu.org/licenses/>. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +<http://www.gnu.org/philosophy/why-not-lgpl.html>. @@ -1 +1,61 @@ -TODO +# SSSync +SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes. + +## What is SSSync? +Simple and Stupid Synchronizer performs one-way synchronisation of any data that follow the "key => attributes => values set" data model. Each synchronisation task could have multiple sources and one destination. The multi-valued data model of SSSync is inspired of the one used for LDAP entries. Any stricter model is usable with SSSync, notably SQL resultset and more generally anything that sounds like a data table. + +### Functionnal facts + - Meant to be started periodically (ie. nightly) + - No daemon, no persistent things, no data change tracking + - Dry-run mode and safeguards (max exec time, max insert/update/delete operations) + - Minimal data interpretation and mapping (SQL already got a marvellous "as" keyword) + - Validate data against schemas, log problems in human-readable form + - Never rewrites an already up-to-date data (master/slave friendly) + - SysAdmin-friendly (crontab-aware, well-defined exit codes, log verbosity) + +### Technical facts + - Structured configuration files with YaML : simple and stupid, like the rest + - No embded script language, no XML, no ORM mappings + - Basically performs one full read of all sources at each run + - Small code base (2k SLOC), low algorithmic complexity, key-sorted reads and comparisons + - Memory footprint is low (< 64 Mio) whatever input data size + - Performs within seconds, throughput typically limited by destination write rate + - Written in Java, uses great libraries like [https://www.unboundid.com/products/ldapsdk Unbound ID's LDAP SDK] + - Could be self-contained in a single directory like portable apps + - Packaging for Linux (.deb, .rpm) and Windows (.msi) are planned + - Will probably never eat up more than 20 Mio of disk + +## Connectors +### Already shipped-in + - OpenLDAP (source/dest) + - MySQL (source) + - Oracle (source) + - Fixed format CSV (source) +### Not yet done connectors + - JDBC writer + - LDIF reader (painfull because they could mix data and instructions) + - Active Directory (because of the lack of real test environment) + - Arbitrary CSV format (lack of formalism implies huge dev/test effort) + +## Limitations (of current release) + - Values are represented and compared as Java Strings (so, UTF-16) + - Don't expect much about binary blobs or non-printable things + - No password hashing utilities (but nobody still store cleartext passwords, do you ?) + - No data manipulation and transformation in SSSync (maybe it's a feature) + +## SSSync through examples +### Medium-sized university LDAP directory + - Context : 10000 people, 1000 groups and structures, 4 different sync tasks + - Main data source : human resources system stored in an Oracle database + - Additionnal sources : 2 CSV (technical accounts, manual corrections) + - Destination : OpenLDAP (master node, with many replicas via syncrepl) + - Full run time including dry-run passes : 20 seconds +(even in september when there is 400 new students and 50 employee updates to sync per night) +<p align="center"> + <img src="http://www.pouzenc.fr/projects/sssync/SSSync_Doc/diagrams/example1_flow.svg" + alt="SSSync process with 3 combined sources : Oracle + 2 CSV, one destination : slapd" + width="50%"/> +</p> + +### Give me more examples +Please conact me if you have more examples to put here. @@ -0,0 +1,8 @@ + +Ne pas publier les commits avant la licence et les copyrights +pulbier sur pouzenc.fr d'abord (quite a fait de l'auto-push vers githud et source-sup) + +Ne pas shipper les libs dans le git. +Ne pas shipper le driver oracle (pas libre) mais donner l'url + +Ecrire la doc utilisateur, écrire la doc développeur et les diagramme d'archi actuels diff --git a/build.xml b/build.xml new file mode 100644 index 0000000..f6ef8be --- /dev/null +++ b/build.xml @@ -0,0 +1,77 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<project default="create_run_jar" name="Create Runnable Jar for Project SSSync with libraries in sub-folder"> + <!-- Hand-written (with help of Eclipse) --> + + <!-- ANT 1.8 annoying warning work-around --> + <presetdef name="javac"> + <javac includeantruntime="false"/> + </presetdef> + + <target name="clean"> + <ant antfile="build.xml" dir="src/main/" inheritAll="false" target="cleanall"/> + <delete file="build/SSSync.jar"/> + <delete file="build/sssync"/> + <delete dir="build/lib"/> + <delete dir="build/conf"/> + <delete dir="build/data"/> + </target> + + <target name="build"> + <ant antfile="build.xml" dir="src/main/" inheritAll="false" target="build"/> + </target> + + <target depends="build" name="create_run_jar"> + <mkdir dir="build"/> + + <jar destfile="build/SSSync.jar"> + <manifest> + <attribute name="Main-Class" value="SSSync"/> + <attribute name="Class-Path" value=". lib/junit.jar lib/org.hamcrest.core_1.1.0.jar lib/guava-16.0.1.jar lib/log4j-1.2.17.jar lib/snakeyaml-1.11.jar lib/commons-csv-1.0-SNAPSHOT.jar lib/ojdbc6.jar lib/mysql-connector-java-5.1.31-bin.jar lib/unboundid-ldapsdk-se.jar"/> + </manifest> + <fileset dir="src/main/bin"> + <exclude name="**/*Test.class"/> + <exclude name="**/*Tests.class"/> + </fileset> + <fileset dir="src/core/bin"> + <exclude name="**/*Test.class"/> + <exclude name="**/*Tests.class"/> + </fileset> + <fileset dir="src/connectors/bin"> + <exclude name="**/*Test.class"/> + <exclude name="**/*Tests.class"/> + </fileset> + </jar> + + <!-- copy doesn't preserve file's permission --> + <copy file="src/main/sssync.sh" tofile="build/sssync"/> + <exec executable="chmod"> + <arg value="a+x"/> + <arg value="build/sssync"/> + </exec> + + <delete dir="build/lib"/> + <mkdir dir="build/lib"/> + <copy file="src/core/lib/guava-16.0.1.jar" todir="build/lib"/> + <copy file="src/main/lib/log4j-1.2.17.jar" todir="build/lib"/> + <copy file="src/main/lib/snakeyaml-1.11.jar" todir="build/lib"/> + <copy file="src/connectors/lib/commons-csv-1.0-SNAPSHOT.jar" todir="build/lib"/> + <copy file="src/connectors/lib/ojdbc6.jar" todir="build/lib"/> + <copy file="src/connectors/lib/mysql-connector-java-5.1.31-bin.jar" todir="build/lib"/> + <copy file="src/connectors/lib/unboundid-ldapsdk-se.jar" todir="build/lib"/> + + <delete dir="build/conf"/> + <mkdir dir="build/conf"/> + <copy file="src/main/conf/connections.yaml" todir="build/conf"/> + <copy file="src/main/conf/sssync.yaml" todir="build/conf"/> + <copy file="src/main/conf/log4j.properties" todir="build/conf"/> + + <mkdir dir="build/conf/queries"/> + <copy file="src/main/conf/queries/people.sql" todir="build/conf/queries"/> + <copy file="src/main/conf/queries/structures.sql" todir="build/conf/queries"/> + + <delete dir="build/data"/> + <mkdir dir="build/data"/> + <copy file="src/main/data/people_append.csv" todir="build/data"/> + <copy file="src/main/data/people_replace.csv" todir="build/data"/> + </target> +</project> diff --git a/doc/.project b/doc/.project new file mode 100644 index 0000000..b68f680 --- /dev/null +++ b/doc/.project @@ -0,0 +1,11 @@ +<?xml version="1.0" encoding="UTF-8"?> +<projectDescription> + <name>SSSync_Doc</name> + <comment></comment> + <projects> + </projects> + <buildSpec> + </buildSpec> + <natures> + </natures> +</projectDescription> diff --git a/doc/diagrams/example1_flow.dia b/doc/diagrams/example1_flow.dia Binary files differnew file mode 100644 index 0000000..66e01f7 --- /dev/null +++ b/doc/diagrams/example1_flow.dia diff --git a/doc/diagrams/example1_flow.svg b/doc/diagrams/example1_flow.svg new file mode 100644 index 0000000..3613ef5 --- /dev/null +++ b/doc/diagrams/example1_flow.svg @@ -0,0 +1,69 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/PR-SVG-20010719/DTD/svg10.dtd"> +<svg width="27cm" height="17cm" viewBox="141 87 534 335" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> + <g> + <path style="fill: #ffffff" d="M 143 102.333 C 156,92.3333 162.5,89 175.5,89 C 188.5,89 195,92.3333 208,102.333 L 208,155.667 C 195,165.667 188.5,169 175.5,169 C 162.5,169 156,165.667 143,155.667 L 143,102.333z"/> + <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 143 102.333 C 156,92.3333 162.5,89 175.5,89 C 188.5,89 195,92.3333 208,102.333 L 208,155.667 C 195,165.667 188.5,169 175.5,169 C 162.5,169 156,165.667 143,155.667 L 143,102.333"/> + <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 143 102.333 C 156,112.333 162.5,115.667 175.5,115.667 C 188.5,115.667 195,112.333 208,102.333"/> + <text font-size="12.7998" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="175.5" y="139.667"> + <tspan x="175.5" y="139.667">Oracle</tspan> + </text> + </g> + <g> + <polygon style="fill: #ffffff" points="143.4,229.4 195.6,229.4 207.6,241.4 207.6,295.4 143.4,295.4 "/> + <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="143.4,229.4 195.6,229.4 207.6,241.4 207.6,295.4 143.4,295.4 "/> + <polyline style="fill: none; fill-opacity:0; stroke-width: 1; stroke: #000000" points="195.6,229.4 195.6,241.4 207.6,241.4 "/> + <text font-size="12.7998" style="fill: #000000;text-anchor:start;font-family:monospace;font-style:normal;font-weight:normal" x="150.4" y="254.3"> + <tspan x="150.4" y="254.3"></tspan> + <tspan x="150.4" y="270.3"> CSV1 </tspan> + <tspan x="150.4" y="286.3"></tspan> + </text> + </g> + <g> + <polygon style="fill: #ffffff" points="143.4,355.8 195.6,355.8 207.6,367.8 207.6,421.8 143.4,421.8 "/> + <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="143.4,355.8 195.6,355.8 207.6,367.8 207.6,421.8 143.4,421.8 "/> + <polyline style="fill: none; fill-opacity:0; stroke-width: 1; stroke: #000000" points="195.6,355.8 195.6,367.8 207.6,367.8 "/> + <text font-size="12.7998" style="fill: #000000;text-anchor:start;font-family:monospace;font-style:normal;font-weight:normal" x="150.4" y="380.7"> + <tspan x="150.4" y="380.7"></tspan> + <tspan x="150.4" y="396.7"> CSV2 </tspan> + <tspan x="150.4" y="412.7"></tspan> + </text> + </g> + <g> + <rect style="fill: #ffffff" x="346" y="231.9" width="118" height="61"/> + <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="346" y="231.9" width="118" height="61"/> + <text font-size="12.7998" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="405" y="266.3"> + <tspan x="405" y="266.3">SSSync</tspan> + </text> + </g> + <g> + <path style="fill: #ffffff" d="M 608.087 231.4 L 663.525,231.4 C 671.841,243.8 674.613,250 674.613,262.4 C 674.613,274.8 671.841,281 663.525,293.4 L 608.087,293.4 C 599.772,281 597,274.8 597,262.4 C 597,250 599.772,243.8 608.087,231.4z"/> + <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 608.087 231.4 L 663.525,231.4 C 671.841,243.8 674.613,250 674.613,262.4 C 674.613,274.8 671.841,281 663.525,293.4 L 608.087,293.4 C 599.772,281 597,274.8 597,262.4 C 597,250 599.772,243.8 608.087,231.4"/> + <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 663.525 231.4 C 655.209,243.8 652.438,250 652.438,262.4 C 652.438,274.8 655.209,281 663.525,293.4"/> + <text font-size="12.7998" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="630.263" y="266.4"> + <tspan x="630.263" y="266.4">slapd</tspan> + </text> + </g> + <g> + <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="208.95" y1="148.443" x2="342.387" y2="226.006"/> + <polygon style="fill: #000000" points="348.872,229.775 337.713,229.072 342.387,226.006 342.739,220.426 "/> + <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="348.872,229.775 337.713,229.072 342.387,226.006 342.739,220.426 "/> + </g> + <g> + <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="208.586" y1="262.4" x2="335.284" y2="262.4"/> + <polygon style="fill: #000000" points="342.784,262.4 332.784,267.4 335.284,262.4 332.784,257.4 "/> + <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="342.784,262.4 332.784,267.4 335.284,262.4 332.784,257.4 "/> + </g> + <g> + <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="208.586" y1="370.578" x2="339.293" y2="298.589"/> + <polygon style="fill: #000000" points="345.862,294.971 339.515,304.175 339.293,298.589 334.691,295.415 "/> + <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="345.862,294.971 339.515,304.175 339.293,298.589 334.691,295.415 "/> + </g> + <g> + <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="474.501" y1="262.4" x2="587.285" y2="262.4"/> + <polygon style="fill: #000000" points="467.001,262.4 477.001,257.4 474.501,262.4 477.001,267.4 "/> + <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="467.001,262.4 477.001,257.4 474.501,262.4 477.001,267.4 "/> + <polygon style="fill: #000000" points="594.785,262.4 584.785,267.4 587.285,262.4 584.785,257.4 "/> + <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="594.785,262.4 584.785,267.4 587.285,262.4 584.785,257.4 "/> + </g> +</svg> diff --git a/doc/screenshots/perf-slapd-localhost.png b/doc/screenshots/perf-slapd-localhost.png Binary files differnew file mode 100644 index 0000000..181bd14 --- /dev/null +++ b/doc/screenshots/perf-slapd-localhost.png diff --git a/src/connectors/.classpath b/src/connectors/.classpath new file mode 100644 index 0000000..e421f8f --- /dev/null +++ b/src/connectors/.classpath @@ -0,0 +1,21 @@ +<?xml version="1.0" encoding="UTF-8"?> +<classpath> + <classpathentry kind="src" path="src"/> + <classpathentry kind="src" path="JUTests"/> + <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-1.6"/> + <classpathentry combineaccessrules="false" kind="src" path="/SSSync_Core"/> + <classpathentry kind="con" path="org.eclipse.jdt.junit.JUNIT_CONTAINER/4"/> + <classpathentry kind="lib" path="lib/commons-csv-1.0-SNAPSHOT.jar"/> + <classpathentry kind="lib" path="lib/ojdbc6.jar"> + <attributes> + <attribute name="javadoc_location" value="jar:platform:/resource/SSSync_Connectors/lib/ojdbc6-javadoc.jar!/"/> + </attributes> + </classpathentry> + <classpathentry kind="lib" path="lib/mysql-connector-java-5.1.31-bin.jar"/> + <classpathentry kind="lib" path="lib/unboundid-ldapsdk-se.jar"> + <attributes> + <attribute name="javadoc_location" value="jar:platform:/resource/SSSync_Connectors/lib/unboundid-ldapsdk-se-javadoc.jar!/"/> + </attributes> + </classpathentry> + <classpathentry kind="output" path="bin"/> +</classpath> diff --git a/src/connectors/.project b/src/connectors/.project new file mode 100644 index 0000000..b4f50df --- /dev/null +++ b/src/connectors/.project @@ -0,0 +1,17 @@ +<?xml version="1.0" encoding="UTF-8"?> +<projectDescription> + <name>SSSync_Connectors</name> + <comment></comment> + <projects> + </projects> + <buildSpec> + <buildCommand> + <name>org.eclipse.jdt.core.javabuilder</name> + <arguments> + </arguments> + </buildCommand> + </buildSpec> + <natures> + <nature>org.eclipse.jdt.core.javanature</nature> + </natures> +</projectDescription> diff --git a/src/connectors/.settings/org.eclipse.jdt.core.prefs b/src/connectors/.settings/org.eclipse.jdt.core.prefs new file mode 100644 index 0000000..8000cd6 --- /dev/null +++ b/src/connectors/.settings/org.eclipse.jdt.core.prefs @@ -0,0 +1,11 @@ +eclipse.preferences.version=1 +org.eclipse.jdt.core.compiler.codegen.inlineJsrBytecode=enabled +org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.6 +org.eclipse.jdt.core.compiler.codegen.unusedLocal=preserve +org.eclipse.jdt.core.compiler.compliance=1.6 +org.eclipse.jdt.core.compiler.debug.lineNumber=generate +org.eclipse.jdt.core.compiler.debug.localVariable=generate +org.eclipse.jdt.core.compiler.debug.sourceFile=generate +org.eclipse.jdt.core.compiler.problem.assertIdentifier=error +org.eclipse.jdt.core.compiler.problem.enumIdentifier=error +org.eclipse.jdt.core.compiler.source=1.6 diff --git a/src/connectors/JUTests/data/io/csv/CSVDataReaderTest.java b/src/connectors/JUTests/data/io/csv/CSVDataReaderTest.java new file mode 100644 index 0000000..6a0e053 --- /dev/null +++ b/src/connectors/JUTests/data/io/csv/CSVDataReaderTest.java @@ -0,0 +1,50 @@ +package data.io.csv; + +import static org.junit.Assert.*; + +import java.io.IOException; +import java.io.StringReader; +import java.util.Iterator; + +import org.junit.Test; + +import data.MVDataEntry; + +public class CSVDataReaderTest { + + + @Test + public void testNext() throws IOException { + CSVDataReader reader = new CSVDataReader( + "testNext", + new StringReader(CSVDataReader.CSV_DEMO), + false + ); + + MVDataEntry expected[] = new MVDataEntry[3]; + expected[0]=new MVDataEntry("line1"); + expected[0].splitAndPut("from", "csv1;csv1bis", ";"); + expected[0].splitAndPut("attr2","csv1",";"); + + expected[1]=new MVDataEntry("line2"); + expected[1].splitAndPut("hello", "all;the;world", ";"); + + expected[2]=new MVDataEntry("line3"); + expected[2].splitAndPut("hello", "all;the;others", ";"); + + // Test twice to check if asking a new iterator "rewinds" correctly + for (int i=0;i<2;i++) { + System.out.println("Loop " + (i+1)); + Iterator<MVDataEntry> readerIt = reader.iterator(); + + for ( MVDataEntry e: expected) { + assertTrue(readerIt.hasNext()); + MVDataEntry r = readerIt.next(); + System.out.println(e + " / " + r); + assertEquals(e, r); + } + assertFalse(readerIt.hasNext()); + } + } + +} diff --git a/src/connectors/JUTests/data/io/ldap/LDAPDataReaderTest.java b/src/connectors/JUTests/data/io/ldap/LDAPDataReaderTest.java new file mode 100644 index 0000000..dcfc602 --- /dev/null +++ b/src/connectors/JUTests/data/io/ldap/LDAPDataReaderTest.java @@ -0,0 +1,94 @@ +package data.io.ldap; + +import static org.junit.Assert.assertTrue; + +import org.junit.Before; +import org.junit.Test; + +import data.MVDataEntry; + +public class LDAPDataReaderTest { + + LDAPConnectionWrapper builder; + + @Before + public void setup() { + builder = new LDAPConnectionWrapper("localhost", 389, "uid=ldapadmin,ou=specialUsers,dc=univ-jfc,dc=fr", "secret"); + } + + /* + @Test + public void testLookAhead1() { + _testLookAhead(1); + } + */ + + @Test + public void testLookAhead16() { + _testLookAhead(16); + } + + @Test + public void testLookAhead32() { + _testLookAhead(32); + } + + @Test + public void testLookAhead64() { + _testLookAhead(64); + } + + @Test + public void testLookAhead128() { + _testLookAhead(128); + } + + @Test + public void testLookAhead192() { + _testLookAhead(192); + } + + @Test + public void testLookAhead256() { + _testLookAhead(256); + } + + @Test + public void testLookAhead512() { + _testLookAhead(512); + } + + @Test + public void testLookAhead1024() { + _testLookAhead(1024); + } + + private void _testLookAhead(int lookAheadAmount) { + System.out.println("_testLookAhead("+lookAheadAmount+")"); + LDAPFlatDataReader reader = builder.newFlatReader("ldap_test", "ou=people,dc=univ-jfc,dc=fr", "uid", lookAheadAmount); + + int resultCount = 0; + String previousKey=null; + for ( MVDataEntry entry : reader ) { + //System.out.println(entry); + if ( previousKey != null ) assertTrue(entry.getKey().compareTo(previousKey) > 0); + resultCount++; + previousKey=entry.getKey(); + } + System.out.println(resultCount); + assertTrue(resultCount>0); + + // Second time with a second iterator (must give the same results) + int newResultCount = 0; + previousKey=null; + for ( MVDataEntry entry : reader ) { + //System.out.println(entry); + if ( previousKey != null ) assertTrue(entry.getKey().compareTo(previousKey) > 0); + newResultCount++; + previousKey=entry.getKey(); + } + System.out.println(newResultCount); + assertTrue(newResultCount == resultCount); + + } +} diff --git a/src/connectors/JUTests/data/io/ldap/LDAPDataWriterTest.java b/src/connectors/JUTests/data/io/ldap/LDAPDataWriterTest.java new file mode 100644 index 0000000..01a8af0 --- /dev/null +++ b/src/connectors/JUTests/data/io/ldap/LDAPDataWriterTest.java @@ -0,0 +1,16 @@ +package data.io.ldap; + +import static org.junit.Assert.*; + +import org.junit.Test; + +public class LDAPDataWriterTest { + + @Test + public void test() { + fail("Not yet implemented"); + } + + // TODO : test update() extensively : null, empty string, add/update/delete subcases... + +} diff --git a/src/connectors/JUTests/data/io/sql/SQLRelDataReaderTest.java b/src/connectors/JUTests/data/io/sql/SQLRelDataReaderTest.java new file mode 100644 index 0000000..a97a98d --- /dev/null +++ b/src/connectors/JUTests/data/io/sql/SQLRelDataReaderTest.java @@ -0,0 +1,115 @@ +package data.io.sql; + +import static org.junit.Assert.*; + +import java.io.File; +import java.io.IOException; +import java.net.URL; + +import org.junit.Before; +import org.junit.Test; + +import data.MVDataEntry; +import data.io.MVDataReader; +import data.io.sql.SQLConnectionWrapper.DBMSType; + + +/* + +CREATE TABLE sssync.people ( + uid CHAR(16) NULL , + uidNumber INT NOT NULL , + gidNumber INT NULL , + cn VARCHAR(45) NULL , + sn VARCHAR(45) NULL , + homeDirectory VARCHAR(45) NULL , + PRIMARY KEY (uid) ); +INSERT INTO sssync.people (uid, uidNumber, gidNumber, cn, sn, homeDirectory) VALUES ('lpouzenc', 1000, 999, 'Ludovic', 'Pouzenc', '/home/lpouzenc'); +INSERT INTO sssync.people (uid, uidNumber, gidNumber, cn, sn, homeDirectory) VALUES ('dpouzenc', 1001, 999, 'Daniel', 'Pouzenc', '/home/dpouzenc'); + + +for i in $(seq 10000 20000); do echo "INSERT INTO sssync.people (uid, uidNumber, gidNumber, cn, sn, homeDirectory) VALUES ('test$i', $i, 999, '$i', 'test', '/home/test$i');"; done | mysql -uroot -p + + + +DROP TABLE IF EXISTS structures; +CREATE TABLE structures ( + supannCodeEntite varchar(15) NOT NULL, + ou varchar(45) NOT NULL, + supannTypeEntite varchar(45) NOT NULL, + supannCodeEntiteParent varchar(45) NOT NULL, + PRIMARY KEY (supannCodeEntite) +) ENGINE=InnoDB DEFAULT CHARSET=latin1; + +INSERT INTO structures VALUES ('2','CUFR','Etablissement','2'),('9','Personnels','Groupe','2'); + + + +TODO : make automated tests with embded Derby base + + */ +public class SQLRelDataReaderTest { + + private static final String TEST_REQUEST = "SELECT p.*, \"person;posixAccount;top\" as objectClass" + + " FROM sssync.people p" + + " ORDER BY 1 ASC;"; + + private SQLConnectionWrapper builder; + private MVDataReader reader1; + private MVDataReader reader2; + + @Before + public void setup() throws IOException { + // Find the folder containing this test class + URL main = SQLRelDataReaderTest.class.getResource("SQLRelDataReaderTest.class"); + if (!"file".equalsIgnoreCase(main.getProtocol())) + throw new IllegalStateException("This class is not stored in a file"); + File currentFolder = new File(main.getPath()).getParentFile(); + + // Build a connection and two readers on it + builder = new SQLConnectionWrapper(DBMSType.mysql, "localhost", 3306, null, "root", "secret", "sssync"); + reader1 = builder.newReader("testMysql1", TEST_REQUEST); + reader2 = builder.newReader("testMysql2", new File(currentFolder, "req_test.sql")); + } + + @Test + public void testNext() { + // First full read on reader1 + int resultCount_r1i1 = 0; + String previousKey_r1i1=null; + for ( MVDataEntry entry : reader1 ) { + //System.out.println(entry); + if ( previousKey_r1i1 != null ) assertTrue(entry.getKey().compareTo(previousKey_r1i1) > 0); + resultCount_r1i1++; + previousKey_r1i1=entry.getKey(); + } + System.out.println(resultCount_r1i1); + assertTrue(resultCount_r1i1 > 0); + + // First half read on reader2 + int resultCount_r2i1 = 0; + String previousKey_r2i1=null; + for ( MVDataEntry entry : reader2 ) { + //System.out.println(entry); + if ( previousKey_r2i1 != null ) assertTrue(entry.getKey().compareTo(previousKey_r2i1) > 0); + resultCount_r2i1++; + previousKey_r2i1=entry.getKey(); + if ( resultCount_r2i1 > resultCount_r1i1 / 2 ) break; + } + System.out.println(resultCount_r2i1); + assertTrue(resultCount_r2i1 > resultCount_r1i1 / 2 ); + + // Second time with a second iterator on reader1 (must give the same results than r1i1) + int resultCount_r1i2 = 0; + String previousKey_r1i2=null; + for ( MVDataEntry entry : reader1 ) { + //System.out.println(entry); + if ( previousKey_r1i2 != null ) assertTrue(entry.getKey().compareTo(previousKey_r1i2) > 0); + resultCount_r1i2++; + previousKey_r1i2=entry.getKey(); + } + System.out.println(resultCount_r1i2); + assertTrue(resultCount_r1i2 == resultCount_r1i1); + } + +} diff --git a/src/connectors/JUTests/data/io/sql/req_test.sql b/src/connectors/JUTests/data/io/sql/req_test.sql new file mode 100644 index 0000000..ab66d5f --- /dev/null +++ b/src/connectors/JUTests/data/io/sql/req_test.sql @@ -0,0 +1,5 @@ +SELECT + p.*, + "person;posixAccount;top" as objectClass +FROM sssync.people p +ORDER BY 1 ASC; diff --git a/src/connectors/build.xml b/src/connectors/build.xml new file mode 100644 index 0000000..fdae9de --- /dev/null +++ b/src/connectors/build.xml @@ -0,0 +1,89 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- WARNING: Eclipse auto-generated file. + Any modifications will be overwritten. + To include a user specific buildfile here, simply create one in the same + directory with the processing instruction <?eclipse.ant.import?> + as the first entry and export the buildfile again. --> +<project basedir="." default="build" name="SSSync_Connectors"> + <property environment="env"/> + <property name="SSSync_Main.location" value="../main"/> + <property name="ECLIPSE_HOME" value="../../../../../../usr/lib/eclipse"/> + <property name="SSSync_Core.location" value="../core"/> + <property name="debuglevel" value="source,lines,vars"/> + <property name="target" value="1.6"/> + <property name="source" value="1.6"/> + <path id="JUnit 4.libraryclasspath"> + <pathelement location="../../../../../../usr/share/eclipse/dropins/jdt/plugins/org.junit_4.8.2.dist/junit.jar"/> + <pathelement location="../../../../../../usr/share/eclipse/dropins/jdt/plugins/org.hamcrest.core_1.1.0.jar"/> + </path> + <path id="SSSync_Core.classpath"> + <pathelement location="${SSSync_Core.location}/bin"/> + <pathelement location="${SSSync_Core.location}/lib/guava-16.0.1.jar"/> + <path refid="JUnit 4.libraryclasspath"/> + </path> + <path id="SSSync_Connectors.classpath"> + <pathelement location="bin"/> + <path refid="SSSync_Core.classpath"/> + <path refid="JUnit 4.libraryclasspath"/> + <pathelement location="lib/commons-csv-1.0-SNAPSHOT.jar"/> + <pathelement location="lib/ojdbc6.jar"/> + <pathelement location="lib/mysql-connector-java-5.1.31-bin.jar"/> + <pathelement location="lib/unboundid-ldapsdk-se.jar"/> + </path> + <target name="init"> + <mkdir dir="bin"/> + <copy includeemptydirs="false" todir="bin"> + <fileset dir="src"> + <exclude name="**/*.java"/> + </fileset> + </copy> + <copy includeemptydirs="false" todir="bin"> + <fileset dir="JUTests"> + <exclude name="**/*.java"/> + </fileset> + </copy> + </target> + <target name="clean"> + <delete dir="bin"/> + </target> + <target depends="clean" name="cleanall"> + <ant antfile="build.xml" dir="${SSSync_Core.location}" inheritAll="false" target="clean"/> + </target> + <target depends="build-subprojects,build-project" name="build"/> + <target name="build-subprojects"> + <ant antfile="build.xml" dir="${SSSync_Core.location}" inheritAll="false" target="build-project"> + <propertyset> + <propertyref name="build.compiler"/> + </propertyset> + </ant> + </target> + <target depends="init" name="build-project"> + <echo message="${ant.project.name}: ${ant.file}"/> + <javac debug="true" debuglevel="${debuglevel}" destdir="bin" includeantruntime="false" source="${source}" target="${target}"> + <src path="src"/> + <src path="JUTests"/> + <classpath refid="SSSync_Connectors.classpath"/> + </javac> + </target> + <target description="Build all projects which reference this project. Useful to propagate changes." name="build-refprojects"> + <ant antfile="build.xml" dir="${SSSync_Main.location}" inheritAll="false" target="clean"/> + <ant antfile="build.xml" dir="${SSSync_Main.location}" inheritAll="false" target="build"> + <propertyset> + <propertyref name="build.compiler"/> + </propertyset> + </ant> + </target> + <target description="copy Eclipse compiler jars to ant lib directory" name="init-eclipse-compiler"> + <copy todir="${ant.library.dir}"> + <fileset dir="${ECLIPSE_HOME}/plugins" includes="org.eclipse.jdt.core_*.jar"/> + </copy> + <unzip dest="${ant.library.dir}"> + <patternset includes="jdtCompilerAdapter.jar"/> + <fileset dir="${ECLIPSE_HOME}/plugins" includes="org.eclipse.jdt.core_*.jar"/> + </unzip> + </target> + <target description="compile project with Eclipse compiler" name="build-eclipse-compiler"> + <property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter"/> + <antcall target="build"/> + </target> +</project> diff --git a/src/connectors/lib/commons-csv-1.0-SNAPSHOT.jar b/src/connectors/lib/commons-csv-1.0-SNAPSHOT.jar Binary files differnew file mode 100644 index 0000000..f6a74f1 --- /dev/null +++ b/src/connectors/lib/commons-csv-1.0-SNAPSHOT.jar diff --git a/src/connectors/lib/derby.jar b/src/connectors/lib/derby.jar Binary files differnew file mode 100644 index 0000000..a4d56f0 --- /dev/null +++ b/src/connectors/lib/derby.jar diff --git a/src/connectors/lib/derbytools.jar b/src/connectors/lib/derbytools.jar Binary files differnew file mode 100644 index 0000000..216ff3e --- /dev/null +++ b/src/connectors/lib/derbytools.jar diff --git a/src/connectors/lib/mysql-connector-java-5.1.31-bin.jar b/src/connectors/lib/mysql-connector-java-5.1.31-bin.jar Binary files differnew file mode 100644 index 0000000..85ae51d --- /dev/null +++ b/src/connectors/lib/mysql-connector-java-5.1.31-bin.jar diff --git a/src/connectors/lib/ojdbc6-javadoc.jar b/src/connectors/lib/ojdbc6-javadoc.jar Binary files differnew file mode 100644 index 0000000..81dfb08 --- /dev/null +++ b/src/connectors/lib/ojdbc6-javadoc.jar diff --git a/src/connectors/lib/ojdbc6.jar b/src/connectors/lib/ojdbc6.jar Binary files differnew file mode 100644 index 0000000..767eba7 --- /dev/null +++ b/src/connectors/lib/ojdbc6.jar diff --git a/src/connectors/lib/orai18n.jar b/src/connectors/lib/orai18n.jar Binary files differnew file mode 100644 index 0000000..9fad382 --- /dev/null +++ b/src/connectors/lib/orai18n.jar diff --git a/src/connectors/lib/unboundid-ldapsdk-se-javadoc.jar b/src/connectors/lib/unboundid-ldapsdk-se-javadoc.jar Binary files differnew file mode 100644 index 0000000..b724779 --- /dev/null +++ b/src/connectors/lib/unboundid-ldapsdk-se-javadoc.jar diff --git a/src/connectors/lib/unboundid-ldapsdk-se.jar b/src/connectors/lib/unboundid-ldapsdk-se.jar Binary files differnew file mode 100644 index 0000000..0932139 --- /dev/null +++ b/src/connectors/lib/unboundid-ldapsdk-se.jar diff --git a/src/connectors/src/data/io/csv/CSVDataReader.java b/src/connectors/src/data/io/csv/CSVDataReader.java new file mode 100644 index 0000000..6dbc8ff --- /dev/null +++ b/src/connectors/src/data/io/csv/CSVDataReader.java @@ -0,0 +1,248 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ +package data.io.csv; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.Reader; +import java.io.StringReader; +import java.util.Iterator; +import java.util.NoSuchElementException; +import java.util.SortedSet; +import java.util.TreeSet; + +import org.apache.commons.csv.CSVFormat; +import org.apache.commons.csv.CSVParser; +import org.apache.commons.csv.CSVRecord; + +import data.MVDataEntry; +import data.io.AbstractMVDataReader; + +/** + * Stream-oriented reader from a particular CSV file. + * Always returns lines/items sorted by lexicographical ascending key. + * + * @author lpouzenc + */ +public class CSVDataReader extends AbstractMVDataReader { + + public static final String CSV_DEMO = + //"key,attr,values\n" + + "line3,hello,all;the;others\n" + + "line1,from,csv1;csv1bis\n" + + "line2,hello,all;the;world\n" + + "line1,attr2,csv1\n" + + ",,\n"; + + public static final CSVFormat DEFAULT_CSV_FORMAT = CSVFormat.EXCEL + .withHeader("key","attr","values") + .withIgnoreSurroundingSpaces(true); + + private final CSVFormat format; + private final Reader dataSourceStream; + + private transient MVDataEntry nextEntry; + private transient CSVRecord nextCSVRecord; + private transient Iterator<CSVRecord> csvIt; + + + /** + * Constructs a CSVDataReader object for parsing a CSV input given via dataSourceStream. + * @param dataSourceName A short string representing this reader (for logging) + * @param dataSourceStream A java.io.Reader from which read the actual CSV data, typically a FileReader + * @param alreadySorted If false, memory cost is around 3 times the CSV file size ! + * @param format Specify the exact format used to encode the CSV file (separators, escaping...) + * @throws IOException + */ + public CSVDataReader(String dataSourceName, Reader dataSourceStream, boolean alreadySorted, CSVFormat format) throws IOException { + this.dataSourceName = dataSourceName; + this.format = format; + + if ( alreadySorted ) { + this.dataSourceStream = dataSourceStream; + } else { + BufferedReader bufReader; + if ( dataSourceStream instanceof BufferedReader ) { + bufReader = (BufferedReader) dataSourceStream; + } else { + bufReader = new BufferedReader(dataSourceStream); + } + this.dataSourceStream = readAndSortLines(bufReader); + } + } + + /** + * Constructs a CSVDataReader object with default CSV format (for CSVParser). + * @param dataSourceName A short string representing this reader (for logging) + * @param dataSourceStream A java.io.Reader from which read the actual CSV data, typically a FileReader + * @param alreadySorted If false, memory cost is around 3 times the CSV file size ! + * @throws IOException + */ + public CSVDataReader(String dataSourceName, Reader dataSourceStream, boolean alreadySorted) throws IOException { + this(dataSourceName, dataSourceStream, alreadySorted, DEFAULT_CSV_FORMAT); + } + + /** + * {@inheritDoc} + * Note : multiple iterators on the same instance are not supported + */ + @Override + public Iterator<MVDataEntry> iterator() { + // When a new iterator is requested, everything should be reset + CSVParser parser; + try { + dataSourceStream.reset(); + parser = new CSVParser(dataSourceStream, format); + } catch (IOException e) { + throw new RuntimeException(e); + } + csvIt = parser.iterator(); + nextCSVRecord = null; + nextEntry = null; + return this; + } + + /** + * {@inheritDoc} + */ + @Override + public boolean hasNext() { + if ( nextEntry == null ) { + lookAhead(); + } + return ( nextEntry != null ); + } + + /** + * {@inheritDoc} + */ + @Override + public MVDataEntry next() { + if ( !hasNext() ) { + throw new NoSuchElementException(); + } + // Pop the lookahead record + MVDataEntry res = nextEntry; + nextEntry=null; + // And return it + return res; + } + + /** + * In-memory File sorting, return as a single String + * @param reader + * @return + * @throws IOException + */ + private Reader readAndSortLines(BufferedReader bufReader) throws IOException { + // Put all the CSV in memory, in a SortedSet + SortedSet<String> lineSet = new TreeSet<String>(); + String inputLine; + int totalCSVSize=0; + while ((inputLine = bufReader.readLine()) != null) { + lineSet.add(inputLine); + totalCSVSize += inputLine.length() + 1; + } + bufReader.close(); // Closes also dataSourceStream + + // Put all sorted lines in a String + StringBuilder allLines = new StringBuilder(totalCSVSize); + for ( String line: lineSet) { + allLines.append(line + "\n"); + } + lineSet = null; // Could help the GC if the input file is huge + + // Build a Java Reader from that String + return new StringReader(allLines.toString()); + } + + /** + * A MVDataEntry could be represented on many CSV lines. + * The key is repeated, the attr could change, the values should change (for given key/attr pair) + */ + private void lookAhead() { + MVDataEntry currEntry = null; + + boolean abort=(nextCSVRecord==null && !csvIt.hasNext()); // Nothing to crunch + boolean done=(nextEntry!=null); // Already looked ahead + while (!abort && !done) { + // Try to get a valid CSVRecord + if ( nextCSVRecord == null ) { + nextCSVRecord = nextValidCSVRecord(); + } + // If no more CSV data + if ( nextCSVRecord == null ) { + // Maybe we have a remaining entry to return + if ( currEntry != null ) { + done=true; continue; + } else { + abort=true; continue; + } + } + + // Now we have a valid CSV line to put in a MVDataEntry + String newKey = nextCSVRecord.get("key"); + + + // If no MVDataEntry yet, it's time to create it (we have data to put into) + if ( currEntry == null ) { + currEntry = new MVDataEntry(newKey); + } + // If CSV line key matches MVDataEntry key, appends attr/values on it + // XXX Tricky code : following condition is always true if the previous one is true + if ( currEntry.getKey().equals(newKey) ) { + currEntry.splitAndPut(nextCSVRecord.get("attr"), nextCSVRecord.get("values"), ";"); + nextCSVRecord = null; // Record consumed + } else { + // Keys are different, we are done (and we have remaining CSV data in nextCSVRecord) + done=true; continue; + } + } + + nextEntry = done?currEntry:null; + } + + /** + * Seek for the next valid record in the CSV file + * @return the next valid CSVRecord + */ + private CSVRecord nextValidCSVRecord() { + CSVRecord res = null; + boolean abort = !csvIt.hasNext(); + boolean done = false; + while (!abort && !done) { + // Try to read a CSV line + res = (csvIt.hasNext())?csvIt.next():null; + + // Break if nothing readable + if ( res == null ) { + abort=true; continue; + } + + // Skip invalid and empty lines + String key = res.get("key"); + if ( key != null && ! key.isEmpty() ) { + done=true; continue; + } + } + + return done?res:null; + } +} diff --git a/src/connectors/src/data/io/ldap/LDAPConnectionWrapper.java b/src/connectors/src/data/io/ldap/LDAPConnectionWrapper.java new file mode 100644 index 0000000..3f6497b --- /dev/null +++ b/src/connectors/src/data/io/ldap/LDAPConnectionWrapper.java @@ -0,0 +1,112 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io.ldap; + +import java.io.Closeable; +import java.io.IOException; + +import com.unboundid.ldap.sdk.BindResult; +import com.unboundid.ldap.sdk.LDAPConnection; +import com.unboundid.ldap.sdk.LDAPConnectionOptions; +import com.unboundid.ldap.sdk.LDAPException; +import com.unboundid.ldap.sdk.ResultCode; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public class LDAPConnectionWrapper implements Closeable { + + private final LDAPConnection conn; + + /** + * TODO javadoc + * @param host + * @param port + * @param bindDN + * @param password + */ + public LDAPConnectionWrapper(String host, int port, String bindDN, String password) { + LDAPConnectionOptions options = new LDAPConnectionOptions(); + options.setAbandonOnTimeout(true); + options.setAllowConcurrentSocketFactoryUse(true); + options.setAutoReconnect(true); + options.setCaptureConnectStackTrace(true); + options.setConnectTimeoutMillis(2000); // 2 seconds + options.setResponseTimeoutMillis(5000); // 5 seconds + options.setUseSynchronousMode(false); + + BindResult bindResult=null; + try { + conn = new LDAPConnection(options, host, port); + bindResult = conn.bind(bindDN, password); + } + catch (LDAPException e) { + throw new RuntimeException(e); + } + + ResultCode resultCode = bindResult.getResultCode(); + if ( resultCode != ResultCode.SUCCESS ) { + throw new RuntimeException("LDAP Bind failed : " + resultCode); + } + } + + /** + * Builds a new reader against current connection and a LDAP baseDN. + * + * @param dataSourceName Short name of this data source (for logging) + * @param baseDN Search base DN (will return childs of this DN) + * @param keyAttr Attribute name that is the primary key of the entry, identifying the entry in a unique manner + * @param lookAheadAmount Grab this amount of entries at once (in memory-sorted, 128 could be great) + * @return A new reader ready to iterate on search results + */ + public LDAPFlatDataReader newFlatReader(String dataSourceName, String baseDN, String keyAttr, int lookAheadAmount) { + try { + return new LDAPFlatDataReader(dataSourceName, conn, baseDN, keyAttr, lookAheadAmount); + } catch (LDAPException e) { + throw new RuntimeException(e); + } + } + + /** + * Builds a new writer that could insert/update/delete entries on a particular LDAP connection and baseDN. + * + * @param baseDN Search base DN (will return childs of this DN) + * @param keyAttr Attribute name that is the primary key of the entry, identifying the entry in a unique manner + * @return A new writter limited on a particular baseDN + */ + public LDAPFlatDataWriter newFlatWriter(String baseDN, String keyAttr) { + try { + return new LDAPFlatDataWriter(conn, baseDN, keyAttr); + } catch (LDAPException e) { + throw new RuntimeException(e); + } + } + + /** + * Close the current ldap connection. + */ + @Override + public void close() throws IOException { + this.conn.close(); + } +} diff --git a/src/connectors/src/data/io/ldap/LDAPFlatDataReader.java b/src/connectors/src/data/io/ldap/LDAPFlatDataReader.java new file mode 100644 index 0000000..2cc79a8 --- /dev/null +++ b/src/connectors/src/data/io/ldap/LDAPFlatDataReader.java @@ -0,0 +1,178 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io.ldap; + +import java.util.Iterator; +import java.util.NoSuchElementException; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; + +import com.unboundid.ldap.sdk.Attribute; +import com.unboundid.ldap.sdk.Filter; +import com.unboundid.ldap.sdk.LDAPConnection; +import com.unboundid.ldap.sdk.LDAPException; +import com.unboundid.ldap.sdk.SearchRequest; +import com.unboundid.ldap.sdk.SearchResult; +import com.unboundid.ldap.sdk.SearchResultEntry; +import com.unboundid.ldap.sdk.SearchResultListener; +import com.unboundid.ldap.sdk.SearchResultReference; +import com.unboundid.ldap.sdk.SearchScope; + +import data.MVDataEntry; +import data.io.AbstractMVDataReader; + +/** + * Stream-oriented reader from a particular LDAP connection + * Always returns lines/items sorted by lexicographical ascending key + * Consistent even if there is a Writer on same LDAP connection (useful for sync) + * + * @author lpouzenc + */ +public class LDAPFlatDataReader extends AbstractMVDataReader { + + private final LDAPConnection conn; + private final String baseDN; + private final String keyAttr; + private final int lookAheadAmount; + private final SortedSet<String> keys; + + private transient Iterator<String> keysItCached; + private transient Iterator<String> keysItConsumed; + private transient SortedMap<String, MVDataEntry> entries; + + // Listener to feed LDAP search result in SortedMap without instantiating a big fat SearchResult + private final SearchResultListener keysReqListener = new SearchResultListener() { + private static final long serialVersionUID = 3364745402521913458L; + + @Override + public void searchEntryReturned(SearchResultEntry searchEntry) { + keys.add(searchEntry.getAttributeValue(keyAttr)); + } + + @Override + public void searchReferenceReturned(SearchResultReference searchReference) { + throw new RuntimeException("Unsupported : search request for all '" + keyAttr + "' has returned at least one reference (excepected : an entry)"); + } + }; + + /** + * Construct a new reader that wrap a particular LDAP search on a given connection + * @param dataSourceName Short name of this data source (for logging) + * @param conn Already initialized LDAP connection where run the search + * @param baseDN Search base DN (will return childs of this DN) + * @param keyAttr Attribute name that is the primary key of the entry, identifying the entry in a unique manner + * @param lookAheadAmount Grab this amount of entries at once (in memory-sorted, 128 could be great) + * @throws LDAPException + */ + public LDAPFlatDataReader(String dataSourceName, LDAPConnection conn, String baseDN, String keyAttr, int lookAheadAmount) throws LDAPException { + this.dataSourceName = dataSourceName; + this.conn = conn; + this.baseDN = baseDN; + this.keyAttr = keyAttr; + this.lookAheadAmount = lookAheadAmount; + + // Grab all the entries' keys from LDAP connection and put them in this.keys + this.keys = new TreeSet<String>(); + SearchRequest keysReq = new SearchRequest(keysReqListener, baseDN, SearchScope.ONE, Filter.create("(objectClass=*)"), keyAttr); + conn.search(keysReq); + } + + /** + * {@inheritDoc} + * Note : multiple iterators on the same instance are not supported + */ + @Override + public Iterator<MVDataEntry> iterator() { + // Reset the search (it uses two different iterators on the same set) + keysItCached = keys.iterator(); + keysItConsumed = keys.iterator(); + entries = new TreeMap<String, MVDataEntry>(); + + return this; + } + + /** + * {@inheritDoc} + */ + @Override + public boolean hasNext() { + return (keysItConsumed==null)?false:keysItConsumed.hasNext(); + } + + /** + * {@inheritDoc} + */ + @Override + public MVDataEntry next() { + String wantedKey = keysItConsumed.next(); + + // Feed the lookAhead buffer if it is empty (and there is more elements to grab) + if ( entries.isEmpty() && keysItCached.hasNext() ) { + lookAhead(lookAheadAmount); + } + + //FIXME : it is possible to have inconsistency between "entries" content and keysIt* values if some entry is deleted since we have read all the keys + + // Pop an entry from the lookAhead buffer + MVDataEntry wantedEntry = entries.remove(wantedKey); + if ( wantedEntry == null ) { + throw new NoSuchElementException(); + } + + return wantedEntry; + } + + /** + * Performs look-ahead of amount entries, using the next sorted keys previously queried. + * @param amount + */ + private void lookAhead(int amount) { + if ( amount < 1 ) { + throw new IllegalArgumentException("LookAhead amount has to be >= 1"); + } + try { + // Build a search that matches "amount" next entries + Filter filter = Filter.createEqualityFilter(keyAttr, keysItCached.next()); + for (int i=0; ( i < amount-1 ) && keysItCached.hasNext(); i++) { + filter = Filter.createORFilter(filter, Filter.createEqualityFilter(keyAttr, keysItCached.next())); + } + SearchRequest searchRequest = new SearchRequest(baseDN, SearchScope.ONE, filter, "*"); + + // XXX Could use a second listener, as for the keys + // Get all this entries in memory, convert them in MVDataEntry beans and store them in a SortedMap + SearchResult search = conn.search(searchRequest); + + for (SearchResultEntry ldapEntry: search.getSearchEntries()) { + String key = ldapEntry.getAttributeValue(keyAttr); + MVDataEntry mvEntry = new MVDataEntry(key); + + for ( Attribute attr : ldapEntry.getAttributes() ) { + mvEntry.put(attr.getName(), attr.getValues()); + } + entries.put(key, mvEntry); + } + } catch (LDAPException e) { + throw new RuntimeException(e); + } + } +} diff --git a/src/connectors/src/data/io/ldap/LDAPFlatDataWriter.java b/src/connectors/src/data/io/ldap/LDAPFlatDataWriter.java new file mode 100644 index 0000000..d1b8918 --- /dev/null +++ b/src/connectors/src/data/io/ldap/LDAPFlatDataWriter.java @@ -0,0 +1,198 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io.ldap; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import com.unboundid.ldap.sdk.Attribute; +import com.unboundid.ldap.sdk.DN; +import com.unboundid.ldap.sdk.DeleteRequest; +import com.unboundid.ldap.sdk.Entry; +import com.unboundid.ldap.sdk.LDAPConnection; +import com.unboundid.ldap.sdk.LDAPException; +import com.unboundid.ldap.sdk.Modification; +import com.unboundid.ldap.sdk.ModificationType; +import com.unboundid.ldap.sdk.ModifyRequest; +import com.unboundid.ldap.sdk.RDN; +import com.unboundid.ldap.sdk.schema.EntryValidator; +import com.unboundid.ldif.LDIFException; + +import data.MVDataEntry; +import data.io.AbstractMVDataWriter; + +/** + * Stream-oriented LDAP writer from a particular LDAP Directory connection. + * + * @author lpouzenc + */ +public class LDAPFlatDataWriter extends AbstractMVDataWriter { + + private final LDAPConnection conn; + private final DN baseDN; + private final String keyAttr; + private final EntryValidator validator; + + /** + * Construct a new writer that could insert/update/delete entries on a particular LDAP connection and baseDN. + * + * @param conn Already initialized LDAP connection where run the search + * @param baseDN Search base DN (will return childs of this DN) + * @param keyAttr Attribute name that is the primary key of the entry, identifying the entry in a unique manner + * @throws LDAPException + */ + public LDAPFlatDataWriter(LDAPConnection conn, String baseDN, String keyAttr) throws LDAPException { + this.conn = conn; + this.baseDN = new DN(baseDN); + this.keyAttr = keyAttr; + this.validator = new EntryValidator(conn.getSchema()); + } + + /** + * {@inheritDoc} + */ + @Override + public void insert(MVDataEntry newEntry) throws LDAPException { + // Build the DN + DN dn = new DN(new RDN(keyAttr, newEntry.getKey()), baseDN); + + // Convert storage objects + Collection<Attribute> attributes = new ArrayList<Attribute>(); + for ( Map.Entry<String, String> entry : newEntry.getAllEntries() ) { + attributes.add(new Attribute(entry.getKey(), entry.getValue())); + } + Entry newLDAPEntry = new Entry(dn, attributes); + + // Add the entry + if ( dryRun ) { + // In dry-run mode, validate the entry + ArrayList<String> invalidReasons = new ArrayList<String>(5); + boolean valid = validator.entryIsValid(newLDAPEntry, invalidReasons); + if ( !valid ) throw new RuntimeException( + "Entry validator has failed to verify this entry :\n" + newLDAPEntry.toLDIFString() + + "Reasons are :\n" + invalidReasons); + } else { + // In real-run mode, insert the entry + try { + conn.add(newLDAPEntry); + } catch (LDAPException e) { + throw new LDAPException(e.getResultCode(), "Error while inserting this entry :\n" + newLDAPEntry.toLDIFString(), e); + } + } + } + + /** + * {@inheritDoc} + */ + @Override + public void update(MVDataEntry updatedEntry, MVDataEntry originalEntry, Set<String> attrToUpdate) throws LDAPException, LDIFException { + // Build the DN + DN dn = new DN(new RDN(keyAttr, updatedEntry.getKey()), baseDN); + + // Convert storage objects + List<Modification> mods = new ArrayList<Modification>(); + for ( String attr : attrToUpdate ) { + Set<String> originalValues = originalEntry.getValues(attr); + Set<String> updatedValues = updatedEntry.getValues(attr); + + Modification modification = null; + + if ( updatedValues.isEmpty() ) { + modification = new Modification(ModificationType.DELETE, attr); + } else { + String[] updatedValuesArr = updatedValues.toArray(new String[0]); + + if ( originalValues.isEmpty() ) { + modification = new Modification(ModificationType.ADD, attr, updatedValuesArr); + } else { + modification = new Modification(ModificationType.REPLACE, attr, updatedValuesArr); + } + } + + mods.add(modification); + } + ModifyRequest modReq = new ModifyRequest(dn, mods); + + // Update the entry + if ( dryRun ) { + // Simulate originalEntry update + Collection<Attribute> attributes = new ArrayList<Attribute>(); + for ( Map.Entry<String, String> entry : originalEntry.getAllEntries() ) { + attributes.add(new Attribute(entry.getKey(), entry.getValue())); + } + Entry originalLDAPEntry = new Entry(dn, attributes); + + // Warning : Unboundid SDK is okay with mandatory attributes with value "" (empty string) + // OpenLDAP do not allow that empty strings in mandatory attributes. + // Empty strings are discarded by MVDataEntry.put() for now. + Entry modifiedLDAPEntry; + try { + modifiedLDAPEntry = Entry.applyModifications(originalLDAPEntry, false, mods); + } catch (LDAPException originalException) { + throw new RuntimeException("Entry update simulation has failed while running applyModifications()\n" + + "original entry : " + originalEntry + "\n" + + "wanted updated entry : " + updatedEntry + "\n" + + "modification request : " + modReq, + originalException); + } + ArrayList<String> invalidReasons = new ArrayList<String>(5); + boolean valid = validator.entryIsValid(modifiedLDAPEntry, invalidReasons); + if ( !valid ) throw new RuntimeException("Entry update simulation has failed while checking entryIsValid()\n" + + "modified entry : " + modifiedLDAPEntry.toLDIFString() + "\n" + + "reasons :" + invalidReasons); + } else { + // In real-run mode, update the entry + try { + conn.modify(modReq); + } catch (LDAPException originalException) { + throw new LDAPException(originalException.getResultCode(), + "Error while updating this entry :\n" + modReq.toLDIFString(), + originalException); + } + } + } + + /** + * {@inheritDoc} + */ + @Override + public void delete(MVDataEntry existingEntry) throws LDAPException { + // Build the DN + DN dn = new DN(new RDN(keyAttr, existingEntry.getKey()), baseDN); + + // Delete the entry + try { + if ( dryRun ) { + //XXX : try to verify the entry existence in dry-run mode ? + } else { + conn.delete(new DeleteRequest(dn)); + } + } catch (LDAPException originalException) { + throw new LDAPException(originalException.getResultCode(), + "Error while deleting this dn : " + dn.toString(), + originalException); + } + } + +} diff --git a/src/connectors/src/data/io/sql/SQLConnectionWrapper.java b/src/connectors/src/data/io/sql/SQLConnectionWrapper.java new file mode 100644 index 0000000..2bab2c8 --- /dev/null +++ b/src/connectors/src/data/io/sql/SQLConnectionWrapper.java @@ -0,0 +1,136 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io.sql; + +import java.io.Closeable; +import java.io.File; +import java.io.IOException; +import java.sql.Connection; +import java.sql.Driver; +import java.sql.DriverManager; +import java.sql.SQLException; + +import data.io.MVDataReader; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public class SQLConnectionWrapper implements Closeable { + + /** + * Enumeration of supported DBMS. Each use a particular JDBC driver. + */ + public enum DBMSType { oracle, mysql/*, derby*/ } + + private final Connection conn; + + /** + * TODO javadoc + * @param dbms + * @param host + * @param port + * @param ress + * @param user + * @param pass + * @param db + */ + public SQLConnectionWrapper(DBMSType dbms, String host, int port, String ress, String user, String pass, String db) { + + String driverClassName=null; + String url; + + switch ( dbms ) { + case oracle: + driverClassName="oracle.jdbc.driver.OracleDriver"; + url="jdbc:oracle:thin:@" + host + ":" + port + ":" + ress + "/" + db; + break; + case mysql: + driverClassName="com.mysql.jdbc.Driver"; + url="jdbc:mysql://" + host + ":" + port + "/" + db; + break; + /* Could be useful with JUnit tests + case derby: + driverClassName="org.apache.derby.jdbc.EmbeddedDriver"; + url="jdbc:derby:" + db; + break; + */ + default: + throw new IllegalArgumentException("Unsupported DBMSType : " + dbms); + } + + try { + @SuppressWarnings("unchecked") + Class<? extends Driver> clazz = (Class<? extends Driver>) Class.forName(driverClassName); + DriverManager.registerDriver(clazz.newInstance()); + } catch (Exception e) { + throw new RuntimeException("Can't load or register JDBC driver for " + dbms + " (" + driverClassName + ")", e); + } + + try { + conn = DriverManager.getConnection(url, user, pass); + } catch (SQLException e) { + throw new RuntimeException("Can't establish database connection (" + url + ")"); + } + } + + /** + * Builds a new reader against current connection and a File containing a SELECT statement. + * @param name + * @param queryFile + * @return + * @throws IOException + */ + public MVDataReader newReader(String name, File queryFile) throws IOException { + return new SQLRelDataReader(name, conn, queryFile); + } + + /** + * Builds a new reader against current connection and a String containing a SELECT statement. + * @param name + * @param query + * @return + * @throws IOException + */ + public MVDataReader newReader(String name, String query) { + return new SQLRelDataReader(name, conn, query); + } + + /** + * Close the current database connection. + */ + @Override + public void close() throws IOException { + try { + conn.close(); + } catch (SQLException e) { + throw new IOException("Exception occured while trying to close the SQL connection", e); + } + } + + /** + * @return the current database connection (useful for JUnit tests) + */ + public Connection getConn() { + return conn; + } +} diff --git a/src/connectors/src/data/io/sql/SQLRelDataReader.java b/src/connectors/src/data/io/sql/SQLRelDataReader.java new file mode 100644 index 0000000..b6355e9 --- /dev/null +++ b/src/connectors/src/data/io/sql/SQLRelDataReader.java @@ -0,0 +1,173 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io.sql; + +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.sql.Connection; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.Iterator; + +import data.MVDataEntry; +import data.io.AbstractMVDataReader; + +/** + * Stream-oriented reader from a particular RDBMS source. + * + * @author lpouzenc + */ +public class SQLRelDataReader extends AbstractMVDataReader { + + private final Connection conn; + private final String request; + + private transient String columnNames[]; + private transient ResultSet rs; + private transient boolean didNext; + private transient boolean hasNext; + + /** + * Build a new reader from an existing connection and a File containing a SELECT statement. + * @param dataSourceName A short string representing this reader (for logging) + * @param conn A pre-established SQL data connection + * @param queryFile An SQL file containing an SQL SELECT statement + * @throws IOException + */ + public SQLRelDataReader(String dataSourceName, Connection conn, File queryFile) throws IOException { + this.dataSourceName = dataSourceName; + this.conn = conn; + this.request = readEntireFile(queryFile); + } + + /** + * Build a new reader from an existing connection and a String containing a SELECT statement. + * @param dataSourceName A short string representing this reader (for logging) + * @param conn A pre-established SQL data connection + * @param query A String containing an SQL SELECT statement + * @throws IOException + */ + public SQLRelDataReader(String dataSourceName, Connection conn, String query) { + this.dataSourceName = dataSourceName; + this.conn = conn; + this.request = query; + } + + /** + * {@inheritDoc} + * Note : multiple iterators on the same instance are not supported + */ + @Override + public Iterator<MVDataEntry> iterator() { + try { + // Reset iterator-related attributes + hasNext = false; + didNext = false; + + // Close and free any previous request result + if ( rs != null ) { + rs.close(); + } + // (Re-)Execute the SQL request + Statement stmt = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY); + rs = stmt.executeQuery(request); + + // Get the column names + ResultSetMetaData rsmd = rs.getMetaData(); + columnNames = new String[rsmd.getColumnCount()]; + for (int i = 0; i < columnNames.length ; i++) { + // Java SQL : all indices starts at 1 (it sucks !) + columnNames[i] = rsmd.getColumnName(i+1); + } + } catch (SQLException e) { + throw new RuntimeException("Could not execute query : " + e.getMessage() + "\n" + request ); + } + + return this; + } + + /** + * {@inheritDoc} + */ + @Override + public boolean hasNext() { + // java.sql.ResultSet don't implement Iterable interface at all + // It's next() don't return anything except hasNext() result but it moves the cursor ! + if (!didNext) { + try { + hasNext = rs.next(); + } catch (SQLException e) { + throw new RuntimeException(e); + } + didNext = true; + } + return hasNext; + } + + /** + * {@inheritDoc} + */ + @Override + public MVDataEntry next() { + MVDataEntry result = null; + try { + if (!didNext) { + rs.next(); + } + didNext = false; + //TODO Instead of always use the first col, user could choose a specific columnName like in LDAP + String key = rs.getString(1); + result = new MVDataEntry(key); + for (int i = 0; i < columnNames.length ; i++) { + // Java SQL : all indices starts at 1 (it sucks !) + result.splitAndPut(columnNames[i], rs.getString(i+1), ";"); // TODO regex should be an option + } + + } catch (SQLException e) { + throw new RuntimeException("Exception while reading next line in SQL resultset", e); + } + + return result; + } + + /** + * Helper function to load and entire file as a String. + * @param file + * @return + * @throws IOException + */ + private static String readEntireFile(File file) throws IOException { + FileReader input = new FileReader(file); + StringBuilder contents = new StringBuilder(); + char[] buffer = new char[4096]; + int read = 0; + do { + contents.append(buffer, 0, read); + read = input.read(buffer); + } while (read >= 0); + input.close(); + + return contents.toString(); + } +} diff --git a/src/core/.classpath b/src/core/.classpath new file mode 100644 index 0000000..f7de406 --- /dev/null +++ b/src/core/.classpath @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="UTF-8"?> +<classpath> + <classpathentry kind="src" path="src"/> + <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-1.6"/> + <classpathentry kind="src" path="JUTests"/> + <classpathentry kind="lib" path="lib/guava-16.0.1.jar"/> + <classpathentry kind="con" path="org.eclipse.jdt.junit.JUNIT_CONTAINER/4"/> + <classpathentry kind="output" path="bin"/> +</classpath> diff --git a/src/core/.project b/src/core/.project new file mode 100644 index 0000000..acda864 --- /dev/null +++ b/src/core/.project @@ -0,0 +1,17 @@ +<?xml version="1.0" encoding="UTF-8"?> +<projectDescription> + <name>SSSync_Core</name> + <comment></comment> + <projects> + </projects> + <buildSpec> + <buildCommand> + <name>org.eclipse.jdt.core.javabuilder</name> + <arguments> + </arguments> + </buildCommand> + </buildSpec> + <natures> + <nature>org.eclipse.jdt.core.javanature</nature> + </natures> +</projectDescription> diff --git a/src/core/.settings/org.eclipse.jdt.core.prefs b/src/core/.settings/org.eclipse.jdt.core.prefs new file mode 100644 index 0000000..8000cd6 --- /dev/null +++ b/src/core/.settings/org.eclipse.jdt.core.prefs @@ -0,0 +1,11 @@ +eclipse.preferences.version=1 +org.eclipse.jdt.core.compiler.codegen.inlineJsrBytecode=enabled +org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.6 +org.eclipse.jdt.core.compiler.codegen.unusedLocal=preserve +org.eclipse.jdt.core.compiler.compliance=1.6 +org.eclipse.jdt.core.compiler.debug.lineNumber=generate +org.eclipse.jdt.core.compiler.debug.localVariable=generate +org.eclipse.jdt.core.compiler.debug.sourceFile=generate +org.eclipse.jdt.core.compiler.problem.assertIdentifier=error +org.eclipse.jdt.core.compiler.problem.enumIdentifier=error +org.eclipse.jdt.core.compiler.source=1.6 diff --git a/src/core/JUTests/data/MVDataEntryTest.java b/src/core/JUTests/data/MVDataEntryTest.java new file mode 100644 index 0000000..19ccb46 --- /dev/null +++ b/src/core/JUTests/data/MVDataEntryTest.java @@ -0,0 +1,93 @@ +package data; + +import static org.junit.Assert.*; + +import org.junit.Test; + +import com.google.common.collect.HashMultimap; + +public class MVDataEntryTest { + + @Test + public void testMVDataEntryStringIntInt() { + String expected = "{key=line1, attrValPairs={k4=[v4], k1=[v1b, v1a, v1c], k2=[v1c]}}"; + + MVDataEntry e1 = new MVDataEntry("line1", 1, 1); + HashMultimap<String, String> e1v = e1.getAttrValPairs(); + e1v.put("k1", "v1a"); + e1v.put("k1", "v1b"); + e1v.put("k1", "v1b"); // Twice, should disappear silently + e1v.put("k1", "v1c"); + e1v.put("k2", "v1c"); + e1v.put("k4", "v4"); + + assertEquals(expected, e1.toString()); + + } + + @Test + public void testMerge() { + // Test data + MVDataEntry e1 = new MVDataEntry("10"); + HashMultimap<String, String> e1v = e1.getAttrValPairs(); + e1v.put("k1", "v1a"); + e1v.put("k1", "v1b"); + e1v.put("k1", "v1c"); + e1v.put("k2", "v2"); + e1v.put("k4", "v4"); + + MVDataEntry e2 = new MVDataEntry("2"); + HashMultimap<String, String> e2v = e2.getAttrValPairs(); + e2v.put("k2", "v2"); + e2v.put("k1", "v1b"); + e2v.put("k3", "v3"); + + MVDataEntry r1 = new MVDataEntry(e1); + r1.mergeValues(true, e2); + assertNotSame(r1, e1); + String expected1 = "{key=10, attrValPairs={k3=[v3], k4=[v4], k1=[v1b, v1a, v1c], k2=[v2]}}"; + assertEquals(expected1, r1.toString()); + + MVDataEntry r2 = new MVDataEntry(e2); + r2.mergeValues(true, e1); + assertNotSame(r2, e2); + String expected2 = "{key=2, attrValPairs={k3=[v3], k4=[v4], k1=[v1b, v1a, v1c], k2=[v2]}}"; + assertEquals(expected2, r2.toString()); + + MVDataEntry r3 = new MVDataEntry(e1); + r3.mergeValues(false, e2); + assertNotSame(r3, e1); + String expected3 = "{key=10, attrValPairs={k3=[v3], k4=[v4], k1=[v1b], k2=[v2]}}"; + //System.out.println(expected3); + //System.out.println(r3.toString()); + assertEquals(expected3, r3.toString()); + + MVDataEntry r4 = new MVDataEntry(e2); + r4.mergeValues(false, e1); + assertNotSame(r4, e1); + String expected4 = "{key=2, attrValPairs={k3=[v3], k4=[v4], k1=[v1b, v1a, v1c], k2=[v2]}}"; + assertEquals(expected4, r4.toString()); + + assertTrue(!r2.equals(r3)); + assertEquals(r2,r4); + } + + @Test + public void testSplitAndPut() { + MVDataEntry r1 = new MVDataEntry("10"); + r1.splitAndPut("k1", "v1a;v1b;v1c", ";"); + r1.put("k2", "v2", null); // splitAndPut does not support null regex anymore, use put() + r1.splitAndPut("k4", "v4", "^$"); + + MVDataEntry expected1 = new MVDataEntry("10"); + HashMultimap<String, String> expected1v = expected1.getAttrValPairs(); + expected1v.put("k1", "v1a"); + expected1v.put("k1", "v1b"); + expected1v.put("k1", "v1c"); + expected1v.put("k2", "v2"); + expected1v.put("k4", "v4"); + + assertEquals(r1,expected1); + } + +} diff --git a/src/core/JUTests/data/io/filters/MVDataCombinerTest.java b/src/core/JUTests/data/io/filters/MVDataCombinerTest.java new file mode 100644 index 0000000..5d32dd8 --- /dev/null +++ b/src/core/JUTests/data/io/filters/MVDataCombinerTest.java @@ -0,0 +1,148 @@ +package data.io.filters; + +import static org.junit.Assert.*; + +import java.util.Iterator; + +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; + +import data.MVDataEntry; +import data.filters.MVDataCombiner; +import data.filters.MVDataCombiner.MVDataCombineMode; +import data.io.MVDataReader; +import data.io.stub.StubDataReader; + +public class MVDataCombinerTest { + + @Rule + public ExpectedException exception = ExpectedException.none(); + + @Test + public void testOutOfOrderCase() { + // Test Data + MVDataEntry e10 = new MVDataEntry("line2"); + e10.getAttrValPairs().put("merge", "e10"); + MVDataEntry e11 = new MVDataEntry("line1"); + e11.getAttrValPairs().put("merge", "e11"); + + MVDataEntry e21 = new MVDataEntry("line2"); + e21.getAttrValPairs().put("merge", "e21"); + + MVDataEntry[][] fakeEntries = new MVDataEntry[][] { + new MVDataEntry[] { e10, e11 }, + new MVDataEntry[] { e21 }, + }; + + MVDataCombineMode mergeModes[] = new MVDataCombineMode[]{ + MVDataCombineMode.PRIMARY_SOURCE, + MVDataCombineMode.MERGE_APPEND, + }; + + // Expected results + MVDataEntry line1 = new MVDataEntry(e10); + line1.mergeValues(true, e21); + + MVDataEntry expected[] = new MVDataEntry[] { + line1, + null /* Should throw UnsupportedOperationException() before comparing */ + }; + + // Test run + exception.expect(UnsupportedOperationException.class); + doCombineTest(expected, fakeEntries, mergeModes); + } + + + @Test + public void testGeneralCase() { + + // Test Data + MVDataEntry e10 = new MVDataEntry("line3"); + e10.getAttrValPairs().put("from1", "e10"); + e10.getAttrValPairs().put("merge", "e10"); + MVDataEntry e11 = new MVDataEntry("line4"); + e11.getAttrValPairs().put("from1", "e11"); + e11.getAttrValPairs().put("merge", "e11"); + + MVDataEntry e20 = new MVDataEntry("line1"); + e20.getAttrValPairs().put("from2", "e20"); + e20.getAttrValPairs().put("merge", "e20"); + MVDataEntry e21 = new MVDataEntry("line2"); + e21.getAttrValPairs().put("from2", "e21"); + e21.getAttrValPairs().put("merge", "e21"); + MVDataEntry e22 = new MVDataEntry("line3"); + e22.getAttrValPairs().put("from2", "e22"); + e22.getAttrValPairs().put("merge", "e22"); + + MVDataEntry e30 = new MVDataEntry("line2"); + e30.getAttrValPairs().put("from3", "e30"); + e30.getAttrValPairs().put("merge", "e30"); + + + MVDataEntry[][] fakeEntries = new MVDataEntry[][] { + new MVDataEntry[] { e10, e11 }, + new MVDataEntry[] { e20, e21, e22 }, + new MVDataEntry[] { e30 }, + }; + + MVDataCombineMode mergeModes[] = new MVDataCombineMode[]{ + MVDataCombineMode.PRIMARY_SOURCE, + MVDataCombineMode.MERGE_REPLACE, + MVDataCombineMode.MERGE_APPEND, + }; + + // Expected results + MVDataEntry line1 = new MVDataEntry(e20); + + MVDataEntry line2 = new MVDataEntry(e21); + line2.mergeValues(true, e30); + + MVDataEntry line3 = new MVDataEntry(e10); + line3.mergeValues(false, e22); + + MVDataEntry line4 = new MVDataEntry(e11); + + MVDataEntry expected[] = new MVDataEntry[] { + line1,line2,line3,line4 + }; + + // Test run + doCombineTest(expected, fakeEntries, mergeModes); + } + + // TODO : test all Combine modes + + /** + * Helper function to factorise Combiner tests. + * @param expected + * @param fakeEntries + * @param mergeModes + */ + public void doCombineTest(MVDataEntry expected[], MVDataEntry[][] fakeEntries, MVDataCombineMode mergeModes[]) { + // Test init + MVDataReader readers[] = new MVDataReader[fakeEntries.length]; + for (int i = 0; i < fakeEntries.length; i++) { + readers[i] = new StubDataReader("fakeReader"+i,fakeEntries[i]); + } + + MVDataCombiner combiner = new MVDataCombiner("combiner", readers, mergeModes); + + // Test twice to check if asking a new iterator "rewinds" correctly + for (int i=0;i<2;i++) { + //System.out.println("Loop " + (i+1)); + + Iterator<MVDataEntry> combinerIt = combiner.iterator(); + for (int j = 0; j < expected.length; j++) { + assertTrue(combinerIt.hasNext()); + MVDataEntry item = combinerIt.next(); + //System.out.println(expected[i]); + //System.out.println(item); + //System.out.println(); + assertEquals(expected[j], item); + } + assertFalse(combinerIt.hasNext()); + } + } +} diff --git a/src/core/build.xml b/src/core/build.xml new file mode 100644 index 0000000..e46c220 --- /dev/null +++ b/src/core/build.xml @@ -0,0 +1,78 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- WARNING: Eclipse auto-generated file. + Any modifications will be overwritten. + To include a user specific buildfile here, simply create one in the same + directory with the processing instruction <?eclipse.ant.import?> + as the first entry and export the buildfile again. --> +<project basedir="." default="build" name="SSSync_Core"> + <property environment="env"/> + <property name="SSSync_Connectors.location" value="../connectors"/> + <property name="SSSync_Main.location" value="../main"/> + <property name="ECLIPSE_HOME" value="../../../../../../usr/lib/eclipse"/> + <property name="debuglevel" value="source,lines,vars"/> + <property name="target" value="1.6"/> + <property name="source" value="1.6"/> + <path id="JUnit 4.libraryclasspath"> + <pathelement location="../../../../../../usr/share/eclipse/dropins/jdt/plugins/org.junit_4.8.2.dist/junit.jar"/> + <pathelement location="../../../../../../usr/share/eclipse/dropins/jdt/plugins/org.hamcrest.core_1.1.0.jar"/> + </path> + <path id="SSSync_Core.classpath"> + <pathelement location="bin"/> + <pathelement location="lib/guava-16.0.1.jar"/> + <path refid="JUnit 4.libraryclasspath"/> + </path> + <target name="init"> + <mkdir dir="bin"/> + <copy includeemptydirs="false" todir="bin"> + <fileset dir="src"> + <exclude name="**/*.java"/> + </fileset> + </copy> + <copy includeemptydirs="false" todir="bin"> + <fileset dir="JUTests"> + <exclude name="**/*.java"/> + </fileset> + </copy> + </target> + <target name="clean"> + <delete dir="bin"/> + </target> + <target depends="clean" name="cleanall"/> + <target depends="build-subprojects,build-project" name="build"/> + <target name="build-subprojects"/> + <target depends="init" name="build-project"> + <echo message="${ant.project.name}: ${ant.file}"/> + <javac debug="true" debuglevel="${debuglevel}" destdir="bin" includeantruntime="false" source="${source}" target="${target}"> + <src path="src"/> + <src path="JUTests"/> + <classpath refid="SSSync_Core.classpath"/> + </javac> + </target> + <target description="Build all projects which reference this project. Useful to propagate changes." name="build-refprojects"> + <ant antfile="build.xml" dir="${SSSync_Connectors.location}" inheritAll="false" target="clean"/> + <ant antfile="build.xml" dir="${SSSync_Connectors.location}" inheritAll="false" target="build"> + <propertyset> + <propertyref name="build.compiler"/> + </propertyset> + </ant> + <ant antfile="build.xml" dir="${SSSync_Main.location}" inheritAll="false" target="clean"/> + <ant antfile="build.xml" dir="${SSSync_Main.location}" inheritAll="false" target="build"> + <propertyset> + <propertyref name="build.compiler"/> + </propertyset> + </ant> + </target> + <target description="copy Eclipse compiler jars to ant lib directory" name="init-eclipse-compiler"> + <copy todir="${ant.library.dir}"> + <fileset dir="${ECLIPSE_HOME}/plugins" includes="org.eclipse.jdt.core_*.jar"/> + </copy> + <unzip dest="${ant.library.dir}"> + <patternset includes="jdtCompilerAdapter.jar"/> + <fileset dir="${ECLIPSE_HOME}/plugins" includes="org.eclipse.jdt.core_*.jar"/> + </unzip> + </target> + <target description="compile project with Eclipse compiler" name="build-eclipse-compiler"> + <property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter"/> + <antcall target="build"/> + </target> +</project> diff --git a/src/core/lib/guava-16.0.1.jar b/src/core/lib/guava-16.0.1.jar Binary files differnew file mode 100644 index 0000000..2c8127d --- /dev/null +++ b/src/core/lib/guava-16.0.1.jar diff --git a/src/core/src/data/MVDataEntry.java b/src/core/src/data/MVDataEntry.java new file mode 100644 index 0000000..f92a141 --- /dev/null +++ b/src/core/src/data/MVDataEntry.java @@ -0,0 +1,238 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data; + +import java.util.HashSet; +import java.util.Map.Entry; +import java.util.Set; + +import com.google.common.collect.HashMultimap; + +/** + * Generic Multi-Valued data type. Each object store a particular entry. + * Semantics are like in LDAP directories : an entry = a key + a set of multi-valued attributes. + * Relational data like in RDMS are more constrained : columns are fixed for an entire table. + * Null and empty string attribute value are silently discarded. + * + * @author lpouzenc + */ +public class MVDataEntry implements Comparable<MVDataEntry> { + + /** + * The key part that identify this particular entry. + */ + private final String key; + /** + * The data part of this particular entry. + */ + private HashMultimap<String,String> attrValPairs; + + // XXX : add an HashMap for meta or constraints ? + + // Constructors + + /** + * Build a fresh empty MVDataEntry. + * @param key Unique key identifying this entry + */ + public MVDataEntry(String key) { + if ( key == null ) { + throw new IllegalArgumentException("key must be non-null"); + } + this.key = key; + this.attrValPairs = HashMultimap.create(); + } + + /** + * Build a fresh empty MVDataEntry with hints about expected attr/values count. + * @param key Unique key identifying this entry + */ + public MVDataEntry(String key, int expectedAttribs, int expectedValuesPerAttrib) { + if ( key == null ) { + throw new IllegalArgumentException("key must be non-null"); + } + this.key = key; + this.attrValPairs = HashMultimap.create(expectedAttribs, expectedValuesPerAttrib); + } + + /** + * Deep copy of an existing MVDataEntry. + * @param key Unique key identifying this entry + */ + public MVDataEntry(final MVDataEntry copyFrom) { + this.key=copyFrom.key; // String is immutable, so ref copy is okay + this.attrValPairs = HashMultimap.create(copyFrom.attrValPairs); + } + + /** + * Proxy function to return all attribute/value pairs. + * One can use read a MVDataEntry without depending on non-standard HashMultimap. + * @return + */ + public Set<Entry<String, String>> getAllEntries() { + return this.attrValPairs.entries(); + } + + /** + * Proxy function to add an attribute/value pair in attrValPairs. + * One can use MVDataEntry without depending on non-standard HashMultimap. + * + * @param attr + * @param value + */ + public void put(String attr, String... values) { + for (String value: values) { + if ( value != null && !value.isEmpty() ) { + this.attrValPairs.put(attr, value); + } + } + } + + /** + * Proxy function to get all values from a particular attribute. + * One can use MVDataEntry without depending on non-standard HashMultimap. + * @param attr + * @return + */ + public Set<String> getValues(String attr) { + return this.attrValPairs.get(attr); + } + + /** + * Helper function to insert multiple values from a single string. + * + * @param attr + * @param value + * @param splitRegex + */ + public void splitAndPut(String attr, String value, String splitRegex) { + if ( value != null ) { + for (String v : value.split(splitRegex)) { + put(attr, v); + } + } + } + + /** + * Helper function to return list of changed attributes. + * Note : this don't keep track of deleted attributes. + * @param original + * @return + */ + public Set<String> getChangedAttributes(MVDataEntry original) { + HashSet<String> result = new HashSet<String>(); + + for (String attr: this.attrValPairs.keySet()) { + Set<String> thisValue = this.attrValPairs.get(attr); + Set<String> originalValue = original.attrValPairs.get(attr); + if ( ! thisValue.equals(originalValue) ) { + result.add(attr); + } + } + + return result; + } + + /** + * Augment this entry with attr/values from other entries. + * @param appendMode Select behavior on an existing attribute : append values or replace them + * @param entries Entries to merge with current entry + */ + public void mergeValues(boolean appendMode, MVDataEntry... entries) { + for(MVDataEntry entry : entries) { + if ( ! appendMode ) { + for (String attr : entry.attrValPairs.keySet()) { + this.attrValPairs.removeAll(attr); + } + } + this.attrValPairs.putAll(entry.attrValPairs); + } + } + + /** + * Check if this entry seems contains useful data. + * @return true if this entry seems contains useful data + */ + public boolean isValid() { + boolean validKey=(this.key != null && this.key.length() > 0 ); + boolean validVal=(this.attrValPairs != null && ! this.attrValPairs.isEmpty()); + + return (validKey && validVal); + } + + /** + * {@inheritDoc} + */ + @Override + public boolean equals(Object obj) { + // Check for self-comparison (compare object references) + if ( this == obj ) { return true; } + // Check non-nullity and type + if ( !( obj instanceof MVDataEntry) ) { return false; } + // Cast safely + MVDataEntry other = (MVDataEntry) obj; + // Check all fields (known to be always non null) + return ( this.key.equals(other.key) && this.attrValPairs.equals(other.attrValPairs) ); + } + + /** + * Compares entries. Ordering of entries is the ordering of their keys. + * (java.lang.String default ordering : lexicographical ascending order) + */ + @Override + public int compareTo(MVDataEntry other) { + return this.key.compareTo(other.key); + } + + /** + * {@inheritDoc} + */ + @Override + public String toString() { + return "{key=" + key + ", attrValPairs=" + attrValPairs.toString() + "}"; + } + + + // Boring accessors + /** + * @return the attrValPairs + */ + public HashMultimap<String, String> getAttrValPairs() { + return attrValPairs; + } + + /** + * @param attrValPairs the attrValPairs to set + */ + public void setAttrValPairs(HashMultimap<String, String> attrValPairs) { + this.attrValPairs = attrValPairs; + } + + /** + * @return the key (guaranteed to be non-null) + */ + public String getKey() { + return key; + } + + + +} diff --git a/src/core/src/data/filters/MVDataCombiner.java b/src/core/src/data/filters/MVDataCombiner.java new file mode 100644 index 0000000..1b2eb3f --- /dev/null +++ b/src/core/src/data/filters/MVDataCombiner.java @@ -0,0 +1,164 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.filters; + +import java.util.Iterator; +import java.util.NoSuchElementException; + +import data.MVDataEntry; +import data.io.AbstractMVDataReader; +import data.io.MVDataReader; + +/** + * Combines arbitrary number of MVData* sources while behaving same as AbstractMVDataReader. + * This could enable a sync implementation to merge multiple sources + * before sync'ing in a transparent manner. + * To prevent memory consumption, this assumes that all sources will be read + * with lexicographical ascending order on the "key" field. + * + * @author lpouzenc + */ +public class MVDataCombiner extends AbstractMVDataReader { + + public enum MVDataCombineMode { PRIMARY_SOURCE, MERGE_APPEND, MERGE_REPLACE, OVERRIDE }; + + private final MVDataReader[] readers; + private final MVDataCombineMode[] mergeModes; + + private transient Iterator<MVDataEntry>[] readerIterators; + private transient MVDataEntry[] lookAheadData; + private transient String lastKey; + + + public MVDataCombiner(String dataSourceName, MVDataReader[] readers, MVDataCombineMode mergeModes[]) { + if ( readers == null || mergeModes == null || (mergeModes.length != readers.length) ) { + throw new IllegalArgumentException("readers and mergeModes arrays should have same size"); + } + if ( ! (mergeModes.length > 0) || mergeModes[0] != MVDataCombineMode.PRIMARY_SOURCE ) { + throw new IllegalArgumentException("MVDataCombiner first mergeModes should always be PRIMARY_SOURCE"); + } + + this.dataSourceName = dataSourceName; + this.readers = readers.clone(); + this.mergeModes = mergeModes.clone(); + } + + /** + * {@inheritDoc} + */ + @Override + @SuppressWarnings("unchecked") /* for new Iterator[...] */ + public Iterator<MVDataEntry> iterator() { + // Be cautious to reset everything + readerIterators = new Iterator[readers.length]; + for (int i=0; i<readers.length;i++) { + readerIterators[i] = readers[i].iterator(); + } + lookAheadData = new MVDataEntry[readers.length]; + lastKey = null; + + return this; + } + + /** + * {@inheritDoc} + */ + @Override + public boolean hasNext() { + for ( MVDataEntry line : lookAheadData ) { + if ( line != null ) { return true; } + } + for ( MVDataReader reader : readers ) { + if ( reader.hasNext() ) { return true; } + } + return false; + } + + /** + * {@inheritDoc} + */ + @Override + public MVDataEntry next() { + + final String currentKey = lookAheadAll(); + + // Check if there was unsorted lines in source data + if ( lastKey != null && (currentKey.compareTo(lastKey) < 0) ) { + //XXX : this is checked here and in SafeDataReader (redundant), but both are optionnal... + throw new UnsupportedOperationException("At least one data source is out of order. " + + "Data sources are excepted to be read sorted by MVDataEntry key (ascending lexicogrpahical order)"); + } + + // Merge all data sources for key currentKey + MVDataEntry result = null; + for ( int i=0; i<lookAheadData.length; i++) { + if ( lookAheadData[i] != null && lookAheadData[i].getKey().equals(currentKey) ) { + if ( result == null ) { + result = lookAheadData[i]; + } else { + //XXX : some items in LDAP could have constrains like : "not multi-valued". Force MERGE_REPLACE mode ? + //FIXME : honor all Combine modes + result.mergeValues( (mergeModes[i] == MVDataCombineMode.MERGE_APPEND ),lookAheadData[i]); + } + lookAheadData[i]=null; // "Pop" the used entry + } + } + + lastKey = currentKey; + + return result; + } + + private String lookAheadAll() { + String minKey=null; + + // Feed the look-ahead buffer (look forward by 1 value for each reader) + for ( int i=0; i<lookAheadData.length; i++) { + if ( lookAheadData[i] == null && readerIterators[i].hasNext() ) { + lookAheadData[i] = readerIterators[i].next(); + } + } + + // Find the least RelData key from look-ahead buffers + for (MVDataEntry entry: lookAheadData) { + if ( entry != null ) { + final String minKeyCandidate = entry.getKey(); + if ( minKey == null || minKey.compareTo(minKeyCandidate) > 0 ) { + minKey = minKeyCandidate; + } + } + } + + // Sanity checks + if ( minKey == null ) { + // Every reader is empty and look-ahead buffer is empty (hasNext() should have said false) + throw new NoSuchElementException(); + } + + return minKey; + } + + // Boring accessors + + public String getLastKey() { + return lastKey; + } +} diff --git a/src/core/src/data/io/AbstractMVDataReader.java b/src/core/src/data/io/AbstractMVDataReader.java new file mode 100644 index 0000000..3e63de1 --- /dev/null +++ b/src/core/src/data/io/AbstractMVDataReader.java @@ -0,0 +1,49 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io; + +/** + * Stream-oriented abstract reader from a particular data source. + * Memory footprint should not depends on readable line count nor next() call count. + * + * @author lpouzenc + */ +public abstract class AbstractMVDataReader implements MVDataReader { + + protected String dataSourceName="(unknown source)"; + + /** + * Not supported (Readers are read-only). + */ + @Override + public void remove() { + throw new UnsupportedOperationException(); + } + + /* (non-Javadoc) + * @see data.io.MVDataReader#getDataSourceName() + */ + @Override + public String getDataSourceName() { + return dataSourceName; + } + +} diff --git a/src/core/src/data/io/AbstractMVDataWriter.java b/src/core/src/data/io/AbstractMVDataWriter.java new file mode 100644 index 0000000..454e8ce --- /dev/null +++ b/src/core/src/data/io/AbstractMVDataWriter.java @@ -0,0 +1,70 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io; + +import java.util.Set; + +import data.MVDataEntry; + +/** + * Stream-oriented abstract writer from a particular data source. + * All derived writers should honor a dry-run mode. + * + * @author lpouzenc + */ +public abstract class AbstractMVDataWriter implements MVDataWriter { + //TODO : not so useful. Interface extraction was not a good idea ? + + /** + * Dry-run mode flag (disabled by default) + */ + protected boolean dryRun=false; + + /* (non-Javadoc) + * @see data.io.MVDataWriter#isDryRun() + */ + public boolean isDryRun() { + return dryRun; + } + + /* (non-Javadoc) + * @see data.io.MVDataWriter#setDryRun(boolean) + */ + public void setDryRun(boolean dryRun) { + this.dryRun = dryRun; + } + + /* (non-Javadoc) + * @see data.io.MVDataWriter#insert(data.MVDataEntry) + */ + @Override + public abstract void insert(MVDataEntry newEntry) throws Exception; + /* (non-Javadoc) + * @see data.io.MVDataWriter#update(data.MVDataEntry, data.MVDataEntry, java.util.Set) + */ + @Override + public abstract void update(MVDataEntry updatedEntry, MVDataEntry originalEntry, Set<String> attrToUpdate) throws Exception; + /* (non-Javadoc) + * @see data.io.MVDataWriter#delete(data.MVDataEntry) + */ + @Override + public abstract void delete(MVDataEntry existingEntry) throws Exception; +} diff --git a/src/core/src/data/io/MVDataReader.java b/src/core/src/data/io/MVDataReader.java new file mode 100644 index 0000000..8a9871a --- /dev/null +++ b/src/core/src/data/io/MVDataReader.java @@ -0,0 +1,39 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io; + +import java.util.Iterator; + +import data.MVDataEntry; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public interface MVDataReader extends Iterator<MVDataEntry>, Iterable<MVDataEntry>{ + + /** + * @return the dataSourceName + */ + public String getDataSourceName(); + +}
\ No newline at end of file diff --git a/src/core/src/data/io/MVDataWriter.java b/src/core/src/data/io/MVDataWriter.java new file mode 100644 index 0000000..2f16fbc --- /dev/null +++ b/src/core/src/data/io/MVDataWriter.java @@ -0,0 +1,45 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io; + +import java.util.Set; + +import data.MVDataEntry; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public interface MVDataWriter { + + public boolean isDryRun(); + public void setDryRun(boolean dryRun); + + public void insert(MVDataEntry newEntry) throws Exception; + + public void update(MVDataEntry updatedEntry, + MVDataEntry originalEntry, Set<String> attrToUpdate) + throws Exception; + + public void delete(MVDataEntry existingEntry) throws Exception; + +}
\ No newline at end of file diff --git a/src/core/src/data/io/stub/StubDataReader.java b/src/core/src/data/io/stub/StubDataReader.java new file mode 100644 index 0000000..ed91267 --- /dev/null +++ b/src/core/src/data/io/stub/StubDataReader.java @@ -0,0 +1,63 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io.stub; + +import java.util.Iterator; +import java.util.NoSuchElementException; + +import data.MVDataEntry; +import data.io.AbstractMVDataReader; + +/** + * Stub reader implementation for automated tests. + * + * @author lpouzenc + */ +public class StubDataReader extends AbstractMVDataReader { + + private final MVDataEntry fakeEntries[]; + private int cursorRead; + + public StubDataReader(String dataSourceName, MVDataEntry[] fakeEntries) { + this.dataSourceName = dataSourceName; + this.fakeEntries = fakeEntries.clone(); + } + + @Override + public Iterator<MVDataEntry> iterator() { + this.cursorRead = 0; + return this; + } + + @Override + public boolean hasNext() { + return cursorRead < fakeEntries.length; + } + + @Override + public MVDataEntry next() { + if ( ! hasNext() ) { + throw new NoSuchElementException(); + } + return fakeEntries[cursorRead++]; + } + +} diff --git a/src/core/src/data/io/stub/StubDataWriter.java b/src/core/src/data/io/stub/StubDataWriter.java new file mode 100644 index 0000000..cd08e77 --- /dev/null +++ b/src/core/src/data/io/stub/StubDataWriter.java @@ -0,0 +1,104 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io.stub; + +import java.util.Set; + +import data.MVDataEntry; +import data.io.AbstractMVDataWriter; + +/** + * Stub writer implementation for automated tests. + * + * @author lpouzenc + */ +public class StubDataWriter extends AbstractMVDataWriter { + + enum OpKind { INSERT, UPDATE, DELETE }; + + private final int maxLogEntries; + + private OpKind opLog[]; + private MVDataEntry opData[]; + private int cursorLog; + + public StubDataWriter(int maxLogEntries) { + this.maxLogEntries = maxLogEntries; + this.opLog = new OpKind[maxLogEntries]; + this.opData = new MVDataEntry[maxLogEntries]; + } + + @Override + public void insert(MVDataEntry newline) { + if ( cursorLog >= maxLogEntries) { + throw new IllegalStateException(); + } + opLog[cursorLog]=OpKind.INSERT; + opData[cursorLog]=newline; + cursorLog++; + } + + @Override + public void update(MVDataEntry updatedLine, MVDataEntry originalLine, Set<String> attrToUpdate) { + if ( cursorLog >= maxLogEntries) { + throw new IllegalStateException(); + } + opLog[cursorLog]=OpKind.UPDATE; + opData[cursorLog]=updatedLine; + cursorLog++; + } + + @Override + public void delete(MVDataEntry existingLine) { + if ( cursorLog >= maxLogEntries) { + throw new IllegalStateException(); + } + opLog[cursorLog]=OpKind.DELETE; + opData[cursorLog]=existingLine; + cursorLog++; + } + + @Override + public String toString() { + StringBuffer buf = new StringBuffer(); + + for (int i = 0; i < cursorLog; i++) { + buf.append(opLog[i] + ": " + opData[i] + "\n"); + } + + return buf.toString(); + } + + /** + * @return the opLog + */ + public OpKind[] getOpLog() { + return opLog.clone(); + } + + /** + * @return the opData + */ + public MVDataEntry[] getOpData() { + return opData.clone(); + } + +} diff --git a/src/core/src/sync/AbstractSyncTask.java b/src/core/src/sync/AbstractSyncTask.java new file mode 100644 index 0000000..e2ae94d --- /dev/null +++ b/src/core/src/sync/AbstractSyncTask.java @@ -0,0 +1,71 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package sync; + +import java.util.concurrent.Callable; + +/** + * Abstract class to define a common base of all kind of synchronization algorithms. + * + * @author lpouzenc + */ +public abstract class AbstractSyncTask implements Callable<Boolean> { + + /** + * Pretty task name to be inserted in log lines + */ + protected String taskName="(unknown task)"; + + /** + * Dry-run mode flag (disabled by default) + */ + protected boolean dryRun=false; + + /** + * Main method that do the actual sync + */ + public abstract Boolean call(); + + + // Boring accessors + + /** + * @return the dryRun mode status (enabled/disabled) + */ + public boolean isDryRun() { + return dryRun; + } + + /** + * @param dryRun the dryRun mode to set (enabled/disabled) + */ + public void setDryRun(boolean dryRun) { + this.dryRun = dryRun; + } + + /** + * @return the taskName + */ + public String getTaskName() { + return taskName; + } + +} diff --git a/src/main/.classpath b/src/main/.classpath new file mode 100644 index 0000000..33bcbdb --- /dev/null +++ b/src/main/.classpath @@ -0,0 +1,16 @@ +<?xml version="1.0" encoding="UTF-8"?> +<classpath> + <classpathentry kind="src" path="src"/> + <classpathentry kind="src" path="JUTests"/> + <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-1.6"/> + <classpathentry kind="con" path="org.eclipse.jdt.junit.JUNIT_CONTAINER/4"/> + <classpathentry combineaccessrules="false" kind="src" path="/SSSync_Core"/> + <classpathentry kind="lib" path="lib/log4j-1.2.17.jar"/> + <classpathentry kind="lib" path="lib/snakeyaml-1.11.jar"> + <attributes> + <attribute name="javadoc_location" value="jar:platform:/resource/SSSync/lib/snakeyaml-1.11-javadoc.jar!/"/> + </attributes> + </classpathentry> + <classpathentry combineaccessrules="false" kind="src" path="/SSSync_Connectors"/> + <classpathentry kind="output" path="bin"/> +</classpath> diff --git a/src/main/.project b/src/main/.project new file mode 100644 index 0000000..33a3a78 --- /dev/null +++ b/src/main/.project @@ -0,0 +1,17 @@ +<?xml version="1.0" encoding="UTF-8"?> +<projectDescription> + <name>SSSync_Main</name> + <comment></comment> + <projects> + </projects> + <buildSpec> + <buildCommand> + <name>org.eclipse.jdt.core.javabuilder</name> + <arguments> + </arguments> + </buildCommand> + </buildSpec> + <natures> + <nature>org.eclipse.jdt.core.javanature</nature> + </natures> +</projectDescription> diff --git a/src/main/.settings/org.eclipse.jdt.core.prefs b/src/main/.settings/org.eclipse.jdt.core.prefs new file mode 100644 index 0000000..8000cd6 --- /dev/null +++ b/src/main/.settings/org.eclipse.jdt.core.prefs @@ -0,0 +1,11 @@ +eclipse.preferences.version=1 +org.eclipse.jdt.core.compiler.codegen.inlineJsrBytecode=enabled +org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.6 +org.eclipse.jdt.core.compiler.codegen.unusedLocal=preserve +org.eclipse.jdt.core.compiler.compliance=1.6 +org.eclipse.jdt.core.compiler.debug.lineNumber=generate +org.eclipse.jdt.core.compiler.debug.localVariable=generate +org.eclipse.jdt.core.compiler.debug.sourceFile=generate +org.eclipse.jdt.core.compiler.problem.assertIdentifier=error +org.eclipse.jdt.core.compiler.problem.enumIdentifier=error +org.eclipse.jdt.core.compiler.source=1.6 diff --git a/src/main/JUTests/AllClientServerTests.java b/src/main/JUTests/AllClientServerTests.java new file mode 100644 index 0000000..cef8ffd --- /dev/null +++ b/src/main/JUTests/AllClientServerTests.java @@ -0,0 +1,18 @@ +import org.junit.runner.RunWith; +import org.junit.runners.Suite; +import org.junit.runners.Suite.SuiteClasses; + +import data.io.ldap.LDAPDataReaderTest; +import data.io.ldap.LDAPDataWriterTest; +import data.io.sql.SQLRelDataReaderTest; + + +@RunWith(Suite.class) +@SuiteClasses({ + // SSSync_Connectors + LDAPDataReaderTest.class, LDAPDataWriterTest.class, + SQLRelDataReaderTest.class, +}) +public class AllClientServerTests { + +} diff --git a/src/main/JUTests/AllLocalTests.java b/src/main/JUTests/AllLocalTests.java new file mode 100644 index 0000000..bc9019d --- /dev/null +++ b/src/main/JUTests/AllLocalTests.java @@ -0,0 +1,29 @@ + + +import org.junit.runner.RunWith; +import org.junit.runners.Suite; +import org.junit.runners.Suite.SuiteClasses; + +import sync.BasicSyncTaskTest; + +import conf.SSSyncConfParserTest; + +import data.MVDataEntryTest; +import data.io.SafeDataReaderTest; +import data.io.csv.CSVDataReaderTest; +import data.io.filters.MVDataCombinerTest; + +@RunWith(Suite.class) +@SuiteClasses( { + // SSSync + SSSyncConfParserTest.class, + SafeDataReaderTest.class, + BasicSyncTaskTest.class, + // SSSync_Connectors (only local) + CSVDataReaderTest.class, + // SSSync_Core + MVDataEntryTest.class, MVDataCombinerTest.class, + } ) +public class AllLocalTests { + +} diff --git a/src/main/JUTests/conf/SSSyncConfParserTest.java b/src/main/JUTests/conf/SSSyncConfParserTest.java new file mode 100644 index 0000000..100df16 --- /dev/null +++ b/src/main/JUTests/conf/SSSyncConfParserTest.java @@ -0,0 +1,69 @@ +package conf; + +import static org.junit.Assert.*; + +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.net.URL; + +import org.junit.Before; +import org.junit.Test; +import org.yaml.snakeyaml.DumperOptions; +import org.yaml.snakeyaml.Yaml; + +public class SSSyncConfParserTest { + + private File currentFolder; + + @Before + public void setup() { + URL main = SSSyncConfParserTest.class.getResource("SSSyncConfParserTest.class"); + if (!"file".equalsIgnoreCase(main.getProtocol())) + throw new IllegalStateException("This class is not stored in a file"); + currentFolder = new File(main.getPath()).getParentFile(); + } + + @Test + public void loadConfigTest() throws Exception { + + String expectedMain = readEntireFile(new File(currentFolder, "testExpectedMain.yaml")); + String expectedConn = readEntireFile(new File(currentFolder, "testExpectedConn.yaml")); + String mainConfigFile = new File(currentFolder, "testMain.yaml").getAbsolutePath(); + String connConfigFile = new File(currentFolder, "testConn.yaml").getAbsolutePath(); + + // Loading (config => beans) + ConfigRootBean confMain = SSSyncConfParser.loadMainConfig(mainConfigFile); + ConfigConnectionsBean confConn = SSSyncConfParser.loadConnConfig(connConfigFile); + + + System.out.println(confMain); + System.out.println(confConn); + + // Dumping (beans => config) + DumperOptions options = new DumperOptions(); + options.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK); + Yaml yamlDump = new Yaml(options); + String dumpMain = yamlDump.dump(confMain); + String dumpConn = yamlDump.dump(confConn); + + // Checking that everything is kept + assertEquals(expectedMain, dumpMain); + assertEquals(expectedConn, dumpConn); + } + + private static String readEntireFile(File file) throws IOException { + FileReader in = new FileReader(file); + StringBuilder contents = new StringBuilder((int) file.length()); + char[] buffer = new char[4096]; + int read = 0; + do { + contents.append(buffer, 0, read); + read = in.read(buffer); + } while (read >= 0); + in.close(); + + return contents.toString(); + } + +} diff --git a/src/main/JUTests/conf/testConn.yaml b/src/main/JUTests/conf/testConn.yaml new file mode 100644 index 0000000..c41063c --- /dev/null +++ b/src/main/JUTests/conf/testConn.yaml @@ -0,0 +1,19 @@ +# This file contains credentials (should be readable only by SSSync) +connections: + - id : ora_1 + type: jdbc + dbms: oracle + ress: gest + host: ora.univ-jfc.fr + port: 1521 + user: GRHUM + pass: secret + db : GHRUM + + - id : ldap_1 + type: ldap + host: localhost + port: 389 + bind: uid=ldapadmin,ou=specialUsers,dc=univ-jfc,dc=fr + pass: secret + diff --git a/src/main/JUTests/conf/testExpectedConn.yaml b/src/main/JUTests/conf/testExpectedConn.yaml new file mode 100644 index 0000000..4cb3421 --- /dev/null +++ b/src/main/JUTests/conf/testExpectedConn.yaml @@ -0,0 +1,22 @@ +!!conf.ConfigConnectionsBean +connections: +- bind: null + db: GHRUM + dbms: oracle + host: ora.univ-jfc.fr + id: ora_1 + pass: secret + port: 1521 + ress: gest + type: jdbc + user: GRHUM +- bind: uid=ldapadmin,ou=specialUsers,dc=univ-jfc,dc=fr + db: null + dbms: null + host: localhost + id: ldap_1 + pass: secret + port: 389 + ress: null + type: ldap + user: null diff --git a/src/main/JUTests/conf/testExpectedMain.yaml b/src/main/JUTests/conf/testExpectedMain.yaml new file mode 100644 index 0000000..dd00aef --- /dev/null +++ b/src/main/JUTests/conf/testExpectedMain.yaml @@ -0,0 +1,70 @@ +!!conf.ConfigRootBean +globals: + maxExecTime: 3 +tasks: +- destination: + attr: uid + base: ou=people,dc=univ-jfc,dc=fr + conn: ldap_1 + kind: ldap + mode: null + name: LDAP de test, ou=people + path: null + query: null + name: People sync + opLimits: + delete: 10 + insert: 100 + update: 10 + skipEntryDelete: false + skipReadErrors: false + sources: + - attr: null + base: null + conn: ora_1 + kind: sql + mode: PRIMARY_SOURCE + name: GHRUM, comptes et personnes + path: null + query: people.sql + - attr: null + base: null + conn: null + kind: csv + mode: MERGE_APPEND + name: CSV personnes additionnelles + path: people_append.csv + query: null + - attr: null + base: null + conn: null + kind: sorted_csv + mode: MERGE_REPLACE + name: CSV correctifs personnes + path: people_replace.csv + query: null +- destination: + attr: supannEntiteAffectation + base: ou=structures,dc=univ-jfc,dc=fr + conn: ldap_1 + kind: ldap + mode: null + name: LDAP de test, ou=structures + path: null + query: null + name: Structure sync + opLimits: + delete: 10 + insert: 10 + update: 10 + skipEntryDelete: true + skipReadErrors: true + sources: + - attr: null + base: null + conn: ora_1 + kind: sql + mode: PRIMARY_SOURCE + name: GHRUM, structures + path: null + query: structures.sql diff --git a/src/main/JUTests/conf/testMain.yaml b/src/main/JUTests/conf/testMain.yaml new file mode 100644 index 0000000..39350b2 --- /dev/null +++ b/src/main/JUTests/conf/testMain.yaml @@ -0,0 +1,54 @@ +# This YAML file describe all synchronization tasks, with their readers and writers + +globals: + maxExecTime: 3 + +tasks: + - name: People sync + opLimits: + insert: 100 + update: 10 + delete: 10 + sources: + - name: GHRUM, comptes et personnes + kind: sql + conn: ora_1 + mode: PRIMARY_SOURCE + query: people.sql + + - name: CSV personnes additionnelles + kind: csv + mode: MERGE_APPEND + path: people_append.csv + + - name: CSV correctifs personnes + kind: sorted_csv + mode: MERGE_REPLACE + path: people_replace.csv + + destination: + name: LDAP de test, ou=people + kind: ldap + conn: ldap_1 + attr: uid + base: ou=people,dc=univ-jfc,dc=fr + + - name: Structure sync + sources: + - name: GHRUM, structures + kind: sql + conn: ora_1 + mode: PRIMARY_SOURCE + query: structures.sql + destination: + name: LDAP de test, ou=structures + kind: ldap + conn: ldap_1 + attr: supannEntiteAffectation + base: ou=structures,dc=univ-jfc,dc=fr + skipEntryDelete: true + skipReadErrors: true + opLimits: + insert: 10 + update: 10 + delete: 10
\ No newline at end of file diff --git a/src/main/JUTests/data/io/SafeDataReaderTest.java b/src/main/JUTests/data/io/SafeDataReaderTest.java new file mode 100644 index 0000000..427004b --- /dev/null +++ b/src/main/JUTests/data/io/SafeDataReaderTest.java @@ -0,0 +1,51 @@ +package data.io; + +import static org.junit.Assert.*; + +import java.util.Iterator; + +import org.apache.log4j.PropertyConfigurator; +import org.junit.BeforeClass; +import org.junit.Test; + +import data.MVDataEntry; +import data.io.stub.StubDataReader; + +public class SafeDataReaderTest { + + private static final String LOG_PROPERTIES_FILE = "conf/log4j.properties"; + + @BeforeClass + public static void setup() { + PropertyConfigurator.configure(LOG_PROPERTIES_FILE); + } + + @Test + public void testNoErrors() { + MVDataEntry testEntries[] = new MVDataEntry[5]; + for (int i=0;i<5;i++) { + testEntries[i] = new MVDataEntry("line"+(i+1)); + testEntries[i].put("attr1", "value"+(i+1)); + } + + StubDataReader src = new StubDataReader("testNoSkipErrors_src", testEntries); + StubDataReader expected = new StubDataReader("testNoSkipErrors_expected", testEntries); + + SafeDataReader reader = new SafeDataReader(src, false); + + // Test twice to check if asking a new iterator "rewinds" correctly + for (int i=0;i<2;i++) { + //System.out.println("Loop " + (i+1)); + Iterator<MVDataEntry> readerIt = reader.iterator(); + for ( MVDataEntry e: expected) { + assertTrue(readerIt.hasNext()); + MVDataEntry r = readerIt.next(); + //System.out.println(e + " / " + r); + assertEquals(e, r); + } + assertFalse(readerIt.hasNext()); + } + } + + //TODO Real tests with messy input readers (null values, exception, hasNext/next() incoherence) +} diff --git a/src/main/JUTests/sync/BasicSyncTaskTest.java b/src/main/JUTests/sync/BasicSyncTaskTest.java new file mode 100644 index 0000000..88d9c98 --- /dev/null +++ b/src/main/JUTests/sync/BasicSyncTaskTest.java @@ -0,0 +1,129 @@ +package sync; + +import static org.junit.Assert.*; + +import java.io.IOException; +import java.io.StringReader; + +import org.apache.log4j.PropertyConfigurator; +import org.junit.BeforeClass; +import org.junit.Test; + +import data.MVDataEntry; +import data.filters.MVDataCombiner; +import data.filters.MVDataCombiner.MVDataCombineMode; +import data.io.MVDataReader; +import data.io.SafeDataReader; +import data.io.csv.CSVDataReader; +import data.io.stub.StubDataReader; +import data.io.stub.StubDataWriter; + +public class BasicSyncTaskTest { + + private static final String LOG_PROPERTIES_FILE = "conf/log4j.properties"; + + @BeforeClass + public static void setup() { + PropertyConfigurator.configure(LOG_PROPERTIES_FILE); + } + + @Test + public void test() throws IOException { + + // Input flows setup + MVDataEntry[] fakeEntries1 = new MVDataEntry[5]; + fakeEntries1[0] = new MVDataEntry("line1"); + fakeEntries1[0].put("hello", "world"); + + fakeEntries1[1] = new MVDataEntry("line2"); + fakeEntries1[1].put("bla", "hidden"); + fakeEntries1[1].put("hello", "merged"); + + fakeEntries1[2] = new MVDataEntry("line3"); + fakeEntries1[2].put("hello", "world"); + + fakeEntries1[3] = new MVDataEntry("line4"); + fakeEntries1[3].put("hello", "world"); + + fakeEntries1[4] = new MVDataEntry("line5"); + fakeEntries1[4].put("hello", "world"); + + + MVDataEntry[] fakeEntries2 = new MVDataEntry[3]; + fakeEntries2[0] = new MVDataEntry("line1"); + fakeEntries2[0].put("hello", "world"); + + fakeEntries2[1] = new MVDataEntry("line2"); + fakeEntries2[1].put("bla", "replaced"); + + fakeEntries2[2] = new MVDataEntry("line3"); + fakeEntries2[2].put("hello", "world"); + + + MVDataEntry[] fakeEntries3 = new MVDataEntry[5]; + fakeEntries3[0] = new MVDataEntry("line2"); + fakeEntries3[0].put("hello", "world"); + fakeEntries3[0].put("extra", "to be preserved"); + + fakeEntries3[1] = new MVDataEntry("line2b"); + fakeEntries3[1].put("to be", "removed", null); + + fakeEntries3[2] = new MVDataEntry("line4"); + fakeEntries3[2].put("hello", "world"); + fakeEntries3[2].put("extra", "to be preserved"); + + fakeEntries3[3] = new MVDataEntry("line5"); + fakeEntries3[3].splitAndPut("hello", "too;much;world", ";"); + + fakeEntries3[4] = new MVDataEntry("line6"); + fakeEntries3[4].put("to be", "removed"); + + StubDataReader fakeReader1 = new StubDataReader("testSrc1", fakeEntries1); + StubDataReader fakeReader2 = new StubDataReader("testSrc3", fakeEntries2); + StubDataReader fakeReader3 = new StubDataReader("testDst", fakeEntries3); + + MVDataReader readers[] = new MVDataReader[]{ + new SafeDataReader(fakeReader1,false), + new SafeDataReader( + new CSVDataReader("testSrc2", + new StringReader(CSVDataReader.CSV_DEMO), + false + ), false + ), + new SafeDataReader(fakeReader2,false), + }; + + MVDataCombineMode mergeModes[] = new MVDataCombineMode[]{ + MVDataCombineMode.PRIMARY_SOURCE, + MVDataCombineMode.MERGE_APPEND, + MVDataCombineMode.MERGE_REPLACE, + }; + + MVDataReader srcReader = new MVDataCombiner("testSrcComb", readers, mergeModes); + MVDataReader dstReader = fakeReader3; + + // Output flow setup + StubDataWriter dstWriter = new StubDataWriter(10); + + // Data sync'er initialization + BasicSyncTask task = new BasicSyncTask("task1", false, srcReader, dstReader, dstWriter); + task.setOperationLimits(10,10,10); + + // Data sync'er run + assertTrue(task.call()); + + // Expected outputs + String expectedDstOps = + "INSERT: {key=line1, attrValPairs={hello=[world], attr2=[csv1], from=[csv1, csv1bis]}}\n" + + "UPDATE: {key=line2, attrValPairs={hello=[the, merged, world, all], bla=[replaced]}}\n" + + "DELETE: {key=line2b, attrValPairs={to be=[removed]}}\n" + + "INSERT: {key=line3, attrValPairs={hello=[world]}}\n" + + // Line 4 must not be updated ! + "UPDATE: {key=line5, attrValPairs={hello=[world]}}\n" + + "DELETE: {key=line6, attrValPairs={to be=[removed]}}\n"; + + // Check results + assertEquals(expectedDstOps, dstWriter.toString()); + } + +} diff --git a/src/main/build.xml b/src/main/build.xml new file mode 100644 index 0000000..8847365 --- /dev/null +++ b/src/main/build.xml @@ -0,0 +1,101 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- WARNING: Eclipse auto-generated file. + Any modifications will be overwritten. + To include a user specific buildfile here, simply create one in the same + directory with the processing instruction <?eclipse.ant.import?> + as the first entry and export the buildfile again. --> +<project basedir="." default="build" name="SSSync_Main"> + <property environment="env"/> + <property name="ECLIPSE_HOME" value="../../../../../../usr/lib/eclipse"/> + <property name="SSSync_Core.location" value="../core"/> + <property name="SSSync_Connectors.location" value="../connectors"/> + <property name="debuglevel" value="source,lines,vars"/> + <property name="target" value="1.6"/> + <property name="source" value="1.6"/> + <path id="JUnit 4.libraryclasspath"> + <pathelement location="../../../../../../usr/share/eclipse/dropins/jdt/plugins/org.junit_4.8.2.dist/junit.jar"/> + <pathelement location="../../../../../../usr/share/eclipse/dropins/jdt/plugins/org.hamcrest.core_1.1.0.jar"/> + </path> + <path id="SSSync_Core.classpath"> + <pathelement location="${SSSync_Core.location}/bin"/> + <pathelement location="${SSSync_Core.location}/lib/guava-16.0.1.jar"/> + <path refid="JUnit 4.libraryclasspath"/> + </path> + <path id="SSSync_Connectors.classpath"> + <pathelement location="${SSSync_Connectors.location}/bin"/> + <path refid="SSSync_Core.classpath"/> + <path refid="JUnit 4.libraryclasspath"/> + <pathelement location="${SSSync_Connectors.location}/lib/commons-csv-1.0-SNAPSHOT.jar"/> + <pathelement location="${SSSync_Connectors.location}/lib/ojdbc6.jar"/> + <pathelement location="${SSSync_Connectors.location}/lib/mysql-connector-java-5.1.31-bin.jar"/> + <pathelement location="${SSSync_Connectors.location}/lib/unboundid-ldapsdk-se.jar"/> + </path> + <path id="SSSync_Main.classpath"> + <pathelement location="bin"/> + <path refid="JUnit 4.libraryclasspath"/> + <path refid="SSSync_Core.classpath"/> + <pathelement location="lib/log4j-1.2.17.jar"/> + <pathelement location="lib/snakeyaml-1.11.jar"/> + <path refid="SSSync_Connectors.classpath"/> + </path> + <target name="init"> + <mkdir dir="bin"/> + <copy includeemptydirs="false" todir="bin"> + <fileset dir="src"> + <exclude name="**/*.java"/> + </fileset> + </copy> + <copy includeemptydirs="false" todir="bin"> + <fileset dir="JUTests"> + <exclude name="**/*.java"/> + </fileset> + </copy> + </target> + <target name="clean"> + <delete dir="bin"/> + </target> + <target depends="clean" name="cleanall"> + <ant antfile="build.xml" dir="${SSSync_Core.location}" inheritAll="false" target="clean"/> + <ant antfile="build.xml" dir="${SSSync_Connectors.location}" inheritAll="false" target="clean"/> + </target> + <target depends="build-subprojects,build-project" name="build"/> + <target name="build-subprojects"> + <ant antfile="build.xml" dir="${SSSync_Core.location}" inheritAll="false" target="build-project"> + <propertyset> + <propertyref name="build.compiler"/> + </propertyset> + </ant> + <ant antfile="build.xml" dir="${SSSync_Connectors.location}" inheritAll="false" target="build-project"> + <propertyset> + <propertyref name="build.compiler"/> + </propertyset> + </ant> + </target> + <target depends="init" name="build-project"> + <echo message="${ant.project.name}: ${ant.file}"/> + <javac debug="true" debuglevel="${debuglevel}" destdir="bin" includeantruntime="false" source="${source}" target="${target}"> + <src path="src"/> + <src path="JUTests"/> + <classpath refid="SSSync_Main.classpath"/> + </javac> + </target> + <target description="Build all projects which reference this project. Useful to propagate changes." name="build-refprojects"/> + <target description="copy Eclipse compiler jars to ant lib directory" name="init-eclipse-compiler"> + <copy todir="${ant.library.dir}"> + <fileset dir="${ECLIPSE_HOME}/plugins" includes="org.eclipse.jdt.core_*.jar"/> + </copy> + <unzip dest="${ant.library.dir}"> + <patternset includes="jdtCompilerAdapter.jar"/> + <fileset dir="${ECLIPSE_HOME}/plugins" includes="org.eclipse.jdt.core_*.jar"/> + </unzip> + </target> + <target description="compile project with Eclipse compiler" name="build-eclipse-compiler"> + <property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter"/> + <antcall target="build"/> + </target> + <target name="SSSync"> + <java classname="SSSync" failonerror="true" fork="yes"> + <classpath refid="SSSync_Main.classpath"/> + </java> + </target> +</project> diff --git a/src/main/conf/connections.yaml b/src/main/conf/connections.yaml new file mode 100644 index 0000000..1918d02 --- /dev/null +++ b/src/main/conf/connections.yaml @@ -0,0 +1,18 @@ +# This file contains credentials (should be readable only by SSSync) +connections: + - id : mysql_1 + type: jdbc + dbms: mysql + host: localhost + port: 3306 + user: root + pass: secret + db : sssync + + - id : ldap_1 + type: ldap + host: localhost + port: 389 + bind: uid=ldapadmin,ou=specialUsers,dc=univ-jfc,dc=fr + pass: secret + diff --git a/src/main/conf/log4j.properties b/src/main/conf/log4j.properties new file mode 100644 index 0000000..6dccde8 --- /dev/null +++ b/src/main/conf/log4j.properties @@ -0,0 +1,29 @@ +# +# our log4j properties / configuration file +# +# STDOUT appender +log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender +log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout +log4j.appender.STDOUT.layout.ConversionPattern=%d %p [%t] %C{1} - %m\n + + +# Normal operation mode +log4j.category.SSSync=INFO, STDOUT +# Configuration dump +#log4j.category.SSSync=DEBUG, STDOUT + +# Normal operation mode (currently nothing more in DEBUG or TRACE modes) +log4j.category.data.io.SafeDataReader=INFO, STDOUT + +# Normal operation mode +#log4j.category.sync.BasicSyncTask=INFO, STDOUT +# Trace insert/update/delete operation +log4j.category.sync.BasicSyncTask=DEBUG, STDOUT +# Trace every key comparison +#log4j.category.sync.BasicSyncTask=TRACE, STDOUT + +# Keep silent about memory and GC +log4j.category.utils.JVMStatsDumper=INFO, STDOUT +# Trace memory usage/GC + dump configuration +#log4j.category.utils.JVMStatsDumper=DEBUG, STDOUT + diff --git a/src/main/conf/queries/people.sql b/src/main/conf/queries/people.sql new file mode 100644 index 0000000..ab66d5f --- /dev/null +++ b/src/main/conf/queries/people.sql @@ -0,0 +1,5 @@ +SELECT + p.*, + "person;posixAccount;top" as objectClass +FROM sssync.people p +ORDER BY 1 ASC; diff --git a/src/main/conf/queries/structures.sql b/src/main/conf/queries/structures.sql new file mode 100644 index 0000000..626273c --- /dev/null +++ b/src/main/conf/queries/structures.sql @@ -0,0 +1,5 @@ +SELECT + s.*, + "supannEntite;organizationalUnit;top" as objectClass +FROM sssync.structures s +ORDER BY 1 ASC; diff --git a/src/main/conf/sssync.yaml b/src/main/conf/sssync.yaml new file mode 100644 index 0000000..b285a37 --- /dev/null +++ b/src/main/conf/sssync.yaml @@ -0,0 +1,56 @@ +# This YAML file describe all synchronization tasks, with their readers and writers + +globals: + maxExecTime: 3 # minutes + +tasks: + - name: People sync + opLimits: + insert: 300 + update: 300 + delete: 300 + sources: + - name: GHRUM, comptes et personnes + kind: sql + conn: mysql_1 + mode: PRIMARY_SOURCE + query: conf/queries/people.sql + + - name: CSV personnes additionnelles + kind: csv + mode: MERGE_APPEND + path: data/people_append.csv + + - name: CSV correctifs personnes + kind: csv + mode: MERGE_REPLACE + path: data/people_replace.csv + + destination: + name: LDAP de test, ou=people + kind: ldap + conn: ldap_1 + attr: uid + base: ou=people,dc=univ-jfc,dc=fr + + - name: Structure sync + opLimits: + insert: 10 + update: 10 + delete: 10 + sources: + - name: GHRUM, structures + kind: sql + conn: mysql_1 + mode: PRIMARY_SOURCE + query: conf/queries/structures.sql + + destination: + name: LDAP de test, ou=structures + kind: ldap + conn: ldap_1 + attr: supannCodeEntite + base: ou=structures,dc=univ-jfc,dc=fr + + skipEntryDelete: true + skipReadErrors: true
\ No newline at end of file diff --git a/src/main/data/people_append.csv b/src/main/data/people_append.csv new file mode 100644 index 0000000..dc526ff --- /dev/null +++ b/src/main/data/people_append.csv @@ -0,0 +1 @@ +lpouzenc,cn,Second-prénom
\ No newline at end of file diff --git a/src/main/data/people_replace.csv b/src/main/data/people_replace.csv new file mode 100644 index 0000000..372ed67 --- /dev/null +++ b/src/main/data/people_replace.csv @@ -0,0 +1,3 @@ +lpouzenc,loginShell,/bin/ksh +,, + diff --git a/src/main/lib/log4j-1.2.17.jar b/src/main/lib/log4j-1.2.17.jar Binary files differnew file mode 100644 index 0000000..068867e --- /dev/null +++ b/src/main/lib/log4j-1.2.17.jar diff --git a/src/main/lib/snakeyaml-1.11-javadoc.jar b/src/main/lib/snakeyaml-1.11-javadoc.jar Binary files differnew file mode 100644 index 0000000..bac2a05 --- /dev/null +++ b/src/main/lib/snakeyaml-1.11-javadoc.jar diff --git a/src/main/lib/snakeyaml-1.11.jar b/src/main/lib/snakeyaml-1.11.jar Binary files differnew file mode 100644 index 0000000..3e237cd --- /dev/null +++ b/src/main/lib/snakeyaml-1.11.jar diff --git a/src/main/src/SSSync.java b/src/main/src/SSSync.java new file mode 100644 index 0000000..422c31e --- /dev/null +++ b/src/main/src/SSSync.java @@ -0,0 +1,208 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +import java.io.IOException; +import java.util.List; +import java.util.concurrent.CancellationException; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import org.apache.log4j.Level; +import org.apache.log4j.Logger; +import org.apache.log4j.PropertyConfigurator; + +import conf.ConfigConnectionsBean; +import conf.ConfigGlobalsBean; +import conf.ConfigRootBean; +import conf.SSSyncConfParser; +import conf.SSSyncConnectionsFactory; +import conf.SSSyncTasksFactory; +import data.io.ConnectionsHolder; + +import sync.BasicSyncTask; +import utils.JVMStatsDumper; + +/** + * Main class for Simple and Stupid Sync'er + * + * @author lpouzenc + */ +public class SSSync { + private static final Logger logger = Logger.getLogger(SSSync.class.getName()); + + private static final String LOG_PROPERTIES_FILE = "conf/log4j.properties"; + private static final String CONFIG_MAIN_FILE = "conf/sssync.yaml"; + private static final String CONFIG_CONN_FILE = "conf/connections.yaml"; + + private static final int ERR_SUCCESS = 0; + private static final int ERR_CONFIG_PARSE_ERROR = 1; + private static final int ERR_CONN_INIT_ERROR = 2; + private static final int ERR_TASK_INIT_ERROR = 3; + private static final int ERR_DRYRUN_FAILURE = 4; + private static final int ERR_REALRUN_FAILURE = 5; + //TODO private static final int ERR_MAXTIME_REACHED = 6; + + /** + * Main entry point. Takes care of cmdline parsing, config files interpretation, + * tasks setup and start. + * + * @param args + */ + public static void main(String[] args) { + // log4j setup (first thing to do) + PropertyConfigurator.configure(LOG_PROPERTIES_FILE); + logger.info("Program start (user: '" + System.getProperty("user.name") + + "', cwd: '" + System.getProperty("user.dir") + "')"); + + //TODO use cmdline args for config file path + String mainConfigFile = CONFIG_MAIN_FILE; + String connConfigFile = CONFIG_CONN_FILE; + + // Config parsing + ConfigRootBean confMain = null; + ConfigConnectionsBean confConn = null; + try { + confMain = SSSyncConfParser.loadMainConfig(mainConfigFile); + confConn = SSSyncConfParser.loadConnConfig(connConfigFile); + } catch (Exception e) { + logger.fatal("Exception while loading configuration", e); + end(ERR_CONFIG_PARSE_ERROR); + } + ConfigGlobalsBean confGlobals = confMain.getGlobals(); + + // Config dump if DEBUG level (or finer) + if ( !logger.getLevel().isGreaterOrEqual(Level.INFO) ) { + logger.debug("Current connection configuration :\n" + confConn); + logger.debug("Current main configuration :\n" + confMain); + } + + // Connections init + logger.info("Connections initialization"); + ConnectionsHolder connections = null; + try { + connections = SSSyncConnectionsFactory.setupConnections(confConn); + } catch (Exception e) { + logger.fatal("Exception while establishing connections", e); + end(ERR_CONN_INIT_ERROR); + } + + // Suggest garbage collector to forget our passwords since we are connected + confConn=null; + System.gc(); + JVMStatsDumper.logMemoryUsage(); + + + // Tasks init + logger.info("Tasks initialization"); + List<BasicSyncTask> tasks = null; + try { + tasks = SSSyncTasksFactory.setupTasks(connections, confMain); + } catch (Exception e) { + logger.fatal("Exception during tasks initialization", e); + end(ERR_TASK_INIT_ERROR); + } + + logger.info("Tasks are ready to start"); + JVMStatsDumper.logMemoryUsage(); + + + // Tasks first (dry) run + if ( ! SSSync.safeTaskRun(tasks, confGlobals.getMaxExecTime(), true) ) { + logger.error("Dry-run pass has shown problems, skipping real synchronization"); + end(ERR_DRYRUN_FAILURE); + } + + // Tasks second (real) run + if ( SSSync.safeTaskRun(tasks, confGlobals.getMaxExecTime(), false) ) { + logger.error("Real-run pass has shown problems, data could be messed up !"); + end(ERR_REALRUN_FAILURE); + } + + // Clean-up + try { + connections.close(); + } catch (IOException e) { + logger.info("Problem during connections closing"); + } + + // Normal exit + end(ERR_SUCCESS); + } + + /** + * Method to run safely a sequence of tasks within a given time period. + * In a separate thread, it runs all the tasks sequentially. + * + * @param list + * @param timeOutInMinute + * @return + * @throws ExecutionException + * @throws InterruptedException + */ + private static boolean safeTaskRun(List<BasicSyncTask> list, long timeOutInMinute, boolean dryRun) { + ExecutorService executor = Executors.newSingleThreadExecutor(); + List<Future<Boolean>> results; + boolean aborted = false; + + logger.info("Starting " + (dryRun?"dry-run":"real-run") + " synchronization pass"); + + for ( BasicSyncTask t : list ) { + t.setDryRun(dryRun); + } + + try { + results = executor.invokeAll(list, timeOutInMinute, TimeUnit.MINUTES); + // Join all tasks, seeking for an unsuccessful execution + for (Future<Boolean> r: results) { + if ( ! r.get() ) { + aborted = true; + } + } + } catch (CancellationException e) { + logger.fatal("Global maximum execution time exhausted, aborting tasks !"); + aborted = true; + } catch (InterruptedException e) { + logger.fatal("Worker thread for task execution was interrupted", e); + aborted = true; + } catch (ExecutionException e) { + logger.error("Exception during tasks execution", e.getCause()); + aborted = true; + } + + JVMStatsDumper.logMemoryUsage(); + executor.shutdown(); + + return !aborted; + } + + /** + * Helper function to always log the end of program + * @param result + */ + private static void end(int result) { + JVMStatsDumper.logGCStats(); + logger.info("Program end (result code: " + result + ")"); + System.exit(result); + } + +} diff --git a/src/main/src/conf/ConfigConnectionBean.java b/src/main/src/conf/ConfigConnectionBean.java new file mode 100644 index 0000000..b43b56f --- /dev/null +++ b/src/main/src/conf/ConfigConnectionBean.java @@ -0,0 +1,111 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import data.io.sql.SQLConnectionWrapper.DBMSType; + +/** + * Generated Configuration Bean + */ +public class ConfigConnectionBean { + + public enum ConnectionType { jdbc, ldap } + + private String id; + private ConnectionType type; + private DBMSType dbms; + private String ress; + private String host; + private int port; + private String user; + private String bind; + private String pass; + private String db; + + public String getId() { + return id; + } + public void setId(String id) { + this.id = id; + } + public ConnectionType getType() { + return type; + } + public void setType(ConnectionType type) { + this.type = type; + } + public DBMSType getDbms() { + return dbms; + } + public void setDbms(DBMSType dbms) { + this.dbms = dbms; + } + public String getRess() { + return ress; + } + public void setRess(String ress) { + this.ress = ress; + } + public String getHost() { + return host; + } + public void setHost(String host) { + this.host = host; + } + public int getPort() { + return port; + } + public void setPort(int port) { + this.port = port; + } + public String getUser() { + return user; + } + public void setUser(String user) { + this.user = user; + } + public String getBind() { + return bind; + } + public void setBind(String bind) { + this.bind = bind; + } + public String getPass() { + return pass; + } + public void setPass(String pass) { + this.pass = pass; + } + public String getDb() { + return db; + } + public void setDb(String db) { + this.db = db; + } + + @Override + public String toString() { + return "ConfigConnectionBean [id=" + id + ", type=" + type + ", dbms=" + dbms + + ", ress=" + ress + ", host=" + host + ", port=" + port + + ", user=" + user + ", bind=" + bind + ", pass=(obfuscated)]"; + } + +} diff --git a/src/main/src/conf/ConfigConnectionsBean.java b/src/main/src/conf/ConfigConnectionsBean.java new file mode 100644 index 0000000..9fb034b --- /dev/null +++ b/src/main/src/conf/ConfigConnectionsBean.java @@ -0,0 +1,45 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import java.util.List; + +/** + * Generated Configuration Bean + */ +public class ConfigConnectionsBean { + + private List<ConfigConnectionBean> connections; + + public List<ConfigConnectionBean> getConnections() { + return connections; + } + + public void setConnections(List<ConfigConnectionBean> connections) { + this.connections = connections; + } + + @Override + public String toString() { + return "ConfigConnectionsBean [connections=" + ConfigRootBean.listDump(connections,1) + "]"; + } + +} diff --git a/src/main/src/conf/ConfigGlobalsBean.java b/src/main/src/conf/ConfigGlobalsBean.java new file mode 100644 index 0000000..256acee --- /dev/null +++ b/src/main/src/conf/ConfigGlobalsBean.java @@ -0,0 +1,41 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +/** + * Generated Configuration Bean + */ +public class ConfigGlobalsBean { + private int maxExecTime; + + public int getMaxExecTime() { + return maxExecTime; + } + + public void setMaxExecTime(int maxExecTime) { + this.maxExecTime = maxExecTime; + } + + @Override + public String toString() { + return "ConfigGlobalsBean [maxExecTime=" + maxExecTime + "]"; + } +} diff --git a/src/main/src/conf/ConfigOpLimitsBean.java b/src/main/src/conf/ConfigOpLimitsBean.java new file mode 100644 index 0000000..8f68e8c --- /dev/null +++ b/src/main/src/conf/ConfigOpLimitsBean.java @@ -0,0 +1,55 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +/** + * Generated Configuration Bean + */ +public class ConfigOpLimitsBean { + private int insert; + private int update; + private int delete; + + public int getInsert() { + return insert; + } + public void setInsert(int insert) { + this.insert = insert; + } + public int getUpdate() { + return update; + } + public void setUpdate(int update) { + this.update = update; + } + public int getDelete() { + return delete; + } + public void setDelete(int delete) { + this.delete = delete; + } + + @Override + public String toString() { + return "ConfigOpLimitsBean [insert=" + insert + ", update=" + update + + ", delete=" + delete + "]"; + } +} diff --git a/src/main/src/conf/ConfigRootBean.java b/src/main/src/conf/ConfigRootBean.java new file mode 100644 index 0000000..acbbd49 --- /dev/null +++ b/src/main/src/conf/ConfigRootBean.java @@ -0,0 +1,73 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import java.util.List; + +/** + * Generated Configuration Bean + */ +public class ConfigRootBean { + + private ConfigGlobalsBean globals; + private List<ConfigTaskBean> tasks; + + public ConfigGlobalsBean getGlobals() { + return globals; + } + public void setGlobals(ConfigGlobalsBean globals) { + this.globals = globals; + } + + public List<ConfigTaskBean> getTasks() { + return tasks; + } + public void setTasks(List<ConfigTaskBean> tasks) { + this.tasks = tasks; + } + + @Override + public String toString() { + return "ConfigRootBean [globals=" + globals + ", tasks=" + listDump(tasks, 1) + "]"; + } + + + public static <T> String listDump(List<T> list, int ident) { + StringBuffer buf = new StringBuffer(); + buf.append('{'); + for (T item : list) { + buf.append('\n'); + for (int i = 0; i < ident; i++) { + buf.append('\t'); + } + buf.append(item.toString()); + buf.append(','); + } + buf.append('\n'); + for (int i = 0; i < ident-1; i++) { + buf.append('\t'); + } + buf.append('}'); + return buf.toString(); + } + + +} diff --git a/src/main/src/conf/ConfigSrcOrDestBean.java b/src/main/src/conf/ConfigSrcOrDestBean.java new file mode 100644 index 0000000..5be1674 --- /dev/null +++ b/src/main/src/conf/ConfigSrcOrDestBean.java @@ -0,0 +1,96 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import data.filters.MVDataCombiner; + +/** + * Generated Configuration Bean + */ +public class ConfigSrcOrDestBean { + + public enum SourceKind { csv, ldap, sorted_csv, sql }; + + private String name; + private SourceKind kind; + private String conn; + private MVDataCombiner.MVDataCombineMode mode; + private String query; + private String path; + private String attr; + private String base; + + public String getName() { + return name; + } + public void setName(String name) { + this.name = name; + } + public SourceKind getKind() { + return kind; + } + public void setKind(SourceKind kind) { + this.kind = kind; + } + public String getConn() { + return conn; + } + public void setConn(String conn) { + this.conn = conn; + } + public MVDataCombiner.MVDataCombineMode getMode() { + return mode; + } + public void setMode(MVDataCombiner.MVDataCombineMode mode) { + this.mode = mode; + } + public String getQuery() { + return query; + } + public void setQuery(String query) { + this.query = query; + } + public String getPath() { + return path; + } + public void setPath(String path) { + this.path = path; + } + public String getAttr() { + return attr; + } + public void setAttr(String attr) { + this.attr = attr; + } + public String getBase() { + return base; + } + public void setBase(String base) { + this.base = base; + } + + @Override + public String toString() { + return "ConfigSrcOrDestBean [name=" + name + ", kind=" + kind + + ", conn=" + conn + ", mode=" + mode + ", query=" + query + + ", path=" + path + ", attr=" + attr + ", base=" + base + "]"; + } +} diff --git a/src/main/src/conf/ConfigTaskBean.java b/src/main/src/conf/ConfigTaskBean.java new file mode 100644 index 0000000..ed34eee --- /dev/null +++ b/src/main/src/conf/ConfigTaskBean.java @@ -0,0 +1,80 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import java.util.List; + +/** + * Generated Configuration Bean + */ +public class ConfigTaskBean { + + private String name; + private ConfigOpLimitsBean opLimits; + private List<ConfigSrcOrDestBean> sources; + private ConfigSrcOrDestBean destination; + private boolean skipReadErrors; + private boolean skipEntryDelete; + + public String getName() { + return name; + } + public void setName(String name) { + this.name = name; + } + public ConfigOpLimitsBean getOpLimits() { + return opLimits; + } + public void setOpLimits(ConfigOpLimitsBean opLimits) { + this.opLimits = opLimits; + } + public List<ConfigSrcOrDestBean> getSources() { + return sources; + } + public void setSources(List<ConfigSrcOrDestBean> sources) { + this.sources = sources; + } + public ConfigSrcOrDestBean getDestination() { + return destination; + } + public void setDestination(ConfigSrcOrDestBean destination) { + this.destination = destination; + } + public boolean isSkipReadErrors() { + return skipReadErrors; + } + public void setSkipReadErrors(boolean skipReadErrors) { + this.skipReadErrors = skipReadErrors; + } + public boolean isSkipEntryDelete() { + return skipEntryDelete; + } + public void setSkipEntryDelete(boolean skipDelete) { + this.skipEntryDelete = skipDelete; + } + @Override + public String toString() { + return "ConfigTaskBean [name=" + name + ", opLimits=" + opLimits + + ", sources=" + sources + ", destination=" + destination + + ", skipReadErrors=" + skipReadErrors + ", skipEntryDelete=" + + skipEntryDelete + "]"; + } +} diff --git a/src/main/src/conf/SSSyncConfParser.java b/src/main/src/conf/SSSyncConfParser.java new file mode 100644 index 0000000..42dc760 --- /dev/null +++ b/src/main/src/conf/SSSyncConfParser.java @@ -0,0 +1,65 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import java.io.FileInputStream; +import java.io.FileNotFoundException; +import java.text.ParseException; + +import org.yaml.snakeyaml.Yaml; +import org.yaml.snakeyaml.constructor.Constructor; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public class SSSyncConfParser { + + public static ConfigRootBean loadMainConfig(String mainConfigFile) throws FileNotFoundException, ParseException { + Yaml yamlMain = new Yaml(new Constructor(ConfigRootBean.class)); + + //TODO : try to prevent weird exceptions when config is not respecting the implicit grammar of the bean tree + + ConfigRootBean confMain = (ConfigRootBean) yamlMain.load(new FileInputStream(mainConfigFile)); + + if ( confMain == null || confMain.getGlobals() == null ) { + throw new ParseException("Config parser has returned a null item", 0); + } + + // TODO : check config sanity and completeness + + return confMain; + } + + public static ConfigConnectionsBean loadConnConfig(String connConfigFile) throws FileNotFoundException, ParseException { + Yaml yamlConn = new Yaml(new Constructor(ConfigConnectionsBean.class)); + + ConfigConnectionsBean confConn = (ConfigConnectionsBean) yamlConn.load(new FileInputStream(connConfigFile)); + + if ( confConn == null ) { + throw new ParseException("Config parser has return a null item", 0); + } + + return confConn; + } + +} diff --git a/src/main/src/conf/SSSyncConnectionsFactory.java b/src/main/src/conf/SSSyncConnectionsFactory.java new file mode 100644 index 0000000..e747258 --- /dev/null +++ b/src/main/src/conf/SSSyncConnectionsFactory.java @@ -0,0 +1,61 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import data.io.ConnectionsHolder; +import data.io.ldap.LDAPConnectionWrapper; +import data.io.sql.SQLConnectionWrapper; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public class SSSyncConnectionsFactory { + + /** + * Setup all connections described in config + * @return + * @throws Exception + */ + public static ConnectionsHolder setupConnections(ConfigConnectionsBean confConn) throws Exception { + ConnectionsHolder connections = new ConnectionsHolder(); + + for ( ConfigConnectionBean conn : confConn.getConnections() ) { + switch (conn.getType()) { + case jdbc: + SQLConnectionWrapper connSQL = new SQLConnectionWrapper(conn.getDbms(), conn.getHost(), conn.getPort(), conn.getRess(), conn.getUser(), conn.getPass(), conn.getDb()); + connections.putConnSQL(conn.getId(), connSQL); + break; + case ldap: + LDAPConnectionWrapper connLDAP = new LDAPConnectionWrapper(conn.getHost(), conn.getPort(), conn.getBind(), conn.getPass()); + connections.putConnLDAP(conn.getId(), connLDAP); + break; + default: + //XXX : find better Exception type + throw new Exception("Bad config : conn '" + conn.getId() + "' unsupported type"); + } + } + + return connections; + } + +} diff --git a/src/main/src/conf/SSSyncTasksFactory.java b/src/main/src/conf/SSSyncTasksFactory.java new file mode 100644 index 0000000..de3e8f6 --- /dev/null +++ b/src/main/src/conf/SSSyncTasksFactory.java @@ -0,0 +1,147 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package conf; + +import java.io.File; +import java.io.FileReader; +import java.util.ArrayList; +import java.util.List; + +import sync.BasicSyncTask; +import data.filters.MVDataCombiner; +import data.filters.MVDataCombiner.MVDataCombineMode; +import data.io.ConnectionsHolder; +import data.io.MVDataReader; +import data.io.MVDataWriter; +import data.io.SafeDataReader; +import data.io.csv.CSVDataReader; +import data.io.ldap.LDAPConnectionWrapper; +import data.io.sql.SQLConnectionWrapper; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public class SSSyncTasksFactory { + + /** + * Build tasks objects with all needed resources from a config beans tree + * @param conf + * @return + * @throws Exception + */ + public static List<BasicSyncTask> setupTasks(ConnectionsHolder connections, ConfigRootBean confMain) throws Exception { + List<BasicSyncTask> tasks = new ArrayList<BasicSyncTask>(); + + // For each task... + for ( ConfigTaskBean confTask: confMain.getTasks() ) { + MVDataReader srcReader=null; + + // Building all sources + + List<ConfigSrcOrDestBean> confSources = confTask.getSources(); + // See if we are in multiple source situation (then MVDataCombiner) or not (then simple MVDataReader) + if ( confSources.size() == 0 ) { + throw new Exception("Bad config : task '" + confTask.getName() + "' has no defined sources"); + } else if ( confSources.size() == 1 ) { + srcReader = new SafeDataReader(_makeReader(connections, confSources.get(0), confTask.getName()), confTask.isSkipReadErrors()); + } else { + List<MVDataReader> readers = new ArrayList<MVDataReader>(); + List<MVDataCombineMode> mergeModes = new ArrayList<MVDataCombineMode>(); + + // For each source of the future MVDataCombiner... + for ( ConfigSrcOrDestBean confSource: confSources ) { + // Create and add the reader and his parameters + readers.add(new SafeDataReader(_makeReader(connections, confSource, confTask.getName()), confTask.isSkipReadErrors())); + mergeModes.add(confSource.getMode()); + } + + srcReader = new MVDataCombiner("srcCombiner", readers.toArray(new MVDataReader[0]), mergeModes.toArray(new MVDataCombineMode[0])); + } + + // Building destination + + MVDataReader dstReader=null; + MVDataWriter dstWriter=null; + + ConfigSrcOrDestBean confDestination = confTask.getDestination(); + switch ( confDestination.getKind() ) { + case ldap: + LDAPConnectionWrapper builder = connections.getLDAPConnectionBuilder(confDestination.getConn()); + // TODO : configurable lookAhead + MVDataReader tmpReader = builder.newFlatReader(confDestination.getName()+"_reader", confDestination.getBase(), confDestination.getAttr(), 128); + dstReader = new SafeDataReader(tmpReader, false); + dstWriter = builder.newFlatWriter(confDestination.getBase(), confDestination.getAttr()); + break; + default: + throw new Exception("Bad config : task '" + confTask.getName() + "' unsupported destination kind"); + } + + // Then building the sync task and add it to the task list + int maxInserts = confTask.getOpLimits().getInsert(); + int maxUpdates = confTask.getOpLimits().getUpdate(); + int maxDeletes = confTask.getOpLimits().getDelete(); + + BasicSyncTask task = new BasicSyncTask(confTask.getName(), false, srcReader, dstReader, dstWriter); + task.setOperationLimits(maxInserts, maxUpdates, maxDeletes); + task.setSkipEntryDelete(confTask.isSkipEntryDelete()); + tasks.add(task); + } + + return tasks; + } + + /** + * Helper function to make a new reader from an existing connection + * @param confSource + * @param taskName + * @return + * @throws Exception + */ + private static MVDataReader _makeReader(ConnectionsHolder connections, ConfigSrcOrDestBean confSource, String taskName) throws Exception { + MVDataReader reader=null; + switch (confSource.getKind()) { + case csv: + reader = new CSVDataReader(confSource.getName(), new FileReader(confSource.getPath()), false); + break; + case ldap: + LDAPConnectionWrapper ldapConnBuilder = connections.getLDAPConnectionBuilder(confSource.getConn()); + //FIXME : if conf error, get...ConnectionBuilder could return null + //TODO : configurable lookAhead + reader = ldapConnBuilder.newFlatReader(confSource.getName(), confSource.getBase(), confSource.getAttr(), 128); + break; + case sorted_csv: + reader = new CSVDataReader(confSource.getName(), new FileReader(confSource.getPath()), true); + break; + case sql: + SQLConnectionWrapper sqlConnBuilder = connections.getSQLConnectionBuilder(confSource.getConn()); + //TODO We assume the query config item is a filepath. It isn't checked anywhere. + reader = sqlConnBuilder.newReader(confSource.getName(), new File(confSource.getQuery())); + break; + default: + throw new Exception("Bad config : task '" + taskName + "' unsupported source kind"); + } + + return reader; + } + +} diff --git a/src/main/src/data/io/ConnectionsHolder.java b/src/main/src/data/io/ConnectionsHolder.java new file mode 100644 index 0000000..3a6e527 --- /dev/null +++ b/src/main/src/data/io/ConnectionsHolder.java @@ -0,0 +1,81 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io; + +import java.io.Closeable; +import java.io.IOException; +import java.util.HashMap; + +import data.io.ldap.LDAPConnectionWrapper; +import data.io.sql.SQLConnectionWrapper; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public class ConnectionsHolder implements Closeable { + + public final HashMap<String, LDAPConnectionWrapper> connMapLDAP; + public final HashMap<String, SQLConnectionWrapper> connMapSQL; + + //TODO : with some refactoring, this class may disappear + /** + * Bean class to keep track of all opened connections in a single object + */ + public ConnectionsHolder() { + this.connMapLDAP = new HashMap<String, LDAPConnectionWrapper>(); + this.connMapSQL = new HashMap<String, SQLConnectionWrapper>(); + } + + public LDAPConnectionWrapper getLDAPConnectionBuilder(String conn) { + return connMapLDAP.get(conn); + } + + public SQLConnectionWrapper getSQLConnectionBuilder(String conn) { + return connMapSQL.get(conn); + } + + public void putConnLDAP(String connId, LDAPConnectionWrapper connLDAP) { + this.connMapLDAP.put(connId, connLDAP); + } + + public void putConnSQL(String connId, SQLConnectionWrapper connSQL) { + this.connMapSQL.put(connId, connSQL); + } + + /** + * Close all connections + */ + @Override + public void close() throws IOException { + // XXX : this will stop at first uncloseable connection. It isn't a very interesting problem however. + for ( LDAPConnectionWrapper connLDAP: connMapLDAP.values() ) { + connLDAP.close(); + } + for ( SQLConnectionWrapper connSQL: connMapSQL.values() ) { + connSQL.close(); + } + } + + + +} diff --git a/src/main/src/data/io/SafeDataReader.java b/src/main/src/data/io/SafeDataReader.java new file mode 100644 index 0000000..2c5dda9 --- /dev/null +++ b/src/main/src/data/io/SafeDataReader.java @@ -0,0 +1,155 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package data.io; + +import java.util.Iterator; +import java.util.NoSuchElementException; + +import org.apache.log4j.Logger; + +import data.MVDataEntry; + +/** + * Multi-valued "safe" stream reader proxy. + * Adds logging and skipReadError mode feature. Check if items are well ordered. + * Ensures consistency of hasNext() / next() even if source stream is faulty. + * Never returns null items but throw NoSuchElementException if no other choices. + * + * @author lpouzenc + */ +public class SafeDataReader extends AbstractMVDataReader { + + private static final Logger logger = Logger.getLogger(SafeDataReader.class.getName()); + + private final MVDataReader src; + /** + * If true, continue even in case of read errors + */ + private final boolean skipReadErrors; + + private transient Iterator<MVDataEntry> srcIt; + private transient boolean abort; + private transient MVDataEntry previousData; + + + public SafeDataReader(MVDataReader src, boolean skipReadErrors) { + this.src = src; + this.dataSourceName = src.getDataSourceName(); + this.skipReadErrors = skipReadErrors; + } + + /** + * {@inheritDoc} + */ + @Override + public Iterator<MVDataEntry> iterator() { + // Reset everything + srcIt = src.iterator(); + abort = false; + previousData = null; + + return this; + } + + /** + * {@inheritDoc} + */ + @Override + public boolean hasNext() { + return (!abort && srcIt.hasNext()); + } + + /** + * {@inheritDoc} + */ + @Override + public MVDataEntry next() { + boolean alreadyWarned=false; + boolean done=false; + MVDataEntry entry = null; + + // Prepare an hint for read exception (knowledge of last successfully read entry could help) + String hint = ( previousData != null )?previousData.getKey():"(nothing)"; + + // Seek for the next valid entry + while (!this.abort && !done && srcIt.hasNext()) { + + // Try to read next entry + try { + entry=src.next(); + if ( entry == null ) throw new NoSuchElementException("Null item returned"); + } catch (Exception e) { + logger.warn(src.getDataSourceName() + " : exception when seeking next valid entry after " + hint, e); + entry = null; // Make sure don't re-use a previous entry + } + + // Sanity checks + boolean valid = ( entry != null && entry.isValid() ); + //XXX Regex should be a parameter + if ( valid && !entry.getKey().matches("^\\p{Print}+$") ) { + logger.warn(src.getDataSourceName() + " : Invalid key found : '" + entry.getKey().replaceAll("[^\\p{Print}]", "?") + "' after " + hint); + valid = false; + } + + + // Two branches : If valid, check ordering then skip or done. If invalid : skip or abort. + if ( valid ) { + // Ensure that data.key is greater than previousData.key or abort + if ( previousData != null && entry.getKey().compareTo(previousData.getKey()) <= 0 ) { + //TODO : this is almost useless in case of reverse-sortered query because everything will be deleted by the Syncer before asking the second item + logger.error(src.getDataSourceName() + " : Input data is not well ordered but the sync task require it : '" + + entry.getKey() + "' is not lexicographically greater than '" + previousData.getKey() + "'"); + // Escape the while loop + abort=true; continue; + } + + // We have found a valid entry, so escape gracefully the loop + done=true; + } else { + // Log read problems and choose between skip or abort + if ( ! this.skipReadErrors ) { + logger.error(src.getDataSourceName() + " has returned an invalid entry after " + hint); + // Escape the while loop + abort=true; continue; + } + if ( !alreadyWarned ) { + alreadyWarned=true; + logger.info("Invalid entry read but skipReadErrors is enabled, will try to read next entry (warned only once)"); + } + + // We don't have a valid entry, give a chance to the next iteration + done=false; + } /* if ( valid )*/ + + } /* while */ + + // If we don't have found anything valid, throw exception (better semantics than returning null) + if (!done) { + throw new NoSuchElementException(); + } + + // Keep track of previous read record + // -> for hinting in log messages when bad things happens + // -> to check if entries are well ordered + previousData=entry; + return entry; + } +} diff --git a/src/main/src/sync/BasicSyncTask.java b/src/main/src/sync/BasicSyncTask.java new file mode 100644 index 0000000..24f34a8 --- /dev/null +++ b/src/main/src/sync/BasicSyncTask.java @@ -0,0 +1,292 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package sync; + + +import java.util.Iterator; +import java.util.NoSuchElementException; +import java.util.Set; + +import org.apache.log4j.Logger; + +import data.MVDataEntry; +import data.io.MVDataReader; +import data.io.MVDataWriter; + +/** + * Basic one-way synchronization code. Uses MVDataEntry semantics. + * Each entry has a key and a set of multi-valued attributes, like LDAP entries. + * Data source is a MVDataReader. Multiple source could be used via MVDataCombiner. + * <br/><br/> + * <b>Warnings :</b> needs MVDataReaders that give key-sorted results. This sync will try + * to delete entries that exists on destination side and don't exist at source side. + * Extra attributes in existing entries on destination side are preserved. + * Look like useful for account's failure password count for instance. + * <br/><br/> + * <b>Notes :</b> Null value and empty strings are not allowed in MVDataEntry, so they are not sync'ed. + * + * @author lpouzenc + */ +public class BasicSyncTask extends AbstractSyncTask { + private static final Logger logger = Logger.getLogger(BasicSyncTask.class.getName()); + + /** + * Source data stream (read-only) + */ + private final MVDataReader srcReader; + /** + * Destination data stream (read) + */ + private final MVDataReader dstReader; + /** + * Destination data stream (write) + */ + private final MVDataWriter dstWriter; + + /** + * If true, disable removal of data on destination side even if detected as obsolete + */ + private boolean skipEntryDelete; + + + private int maxInserts; + private int maxUpdates; + private int maxDeletes; + + private transient int curInserts; + private transient int curUpdates; + private transient int curDeletes; + + + /** + * BasicSyncTask constructor + * Assumes that the *Readers have iterators that returns entries sorted by lexicographical ascending key + * @param taskName Friendly name of this task (for tracing in log files) + * @param srcReader Source data stream (read-only) + * @param dstReader Destination data stream (read) + * @param dstWriter Destination data stream (write) + */ + public BasicSyncTask(String taskName, boolean skipDelete, MVDataReader srcReader, MVDataReader dstReader, MVDataWriter dstWriter) { + this.taskName = taskName; + this.srcReader = srcReader; + this.dstReader = dstReader; + this.dstWriter = dstWriter; + + this.maxInserts = 0; + this.maxUpdates = 0; + this.maxDeletes = 0; + } + + public Boolean call() { + logger.info("task " + taskName + " : starting " + (dryRun?"dry-run":"real") + " pass"); + // Better stack traces "call()" don't say "what" + boolean success = syncTaskRun(); + logger.info("task " + taskName + " : " + (success?"terminated successfully":"aborted")); + + return success; + } + + private boolean syncTaskRun() { + curInserts=0; + curUpdates=0; + curDeletes=0; + dstWriter.setDryRun(dryRun); + + Iterator<MVDataEntry> itSrc = srcReader.iterator(); + Iterator<MVDataEntry> itDst = dstReader.iterator(); + MVDataEntry src = null, dst = null; + boolean srcExhausted = false; + boolean dstExhausted = false; + boolean abort = false; + boolean done = false; + while ( !abort && !done ) { + + // Look-ahead srcReader if previous has been "poped" (or if never read yet) + if ( src == null ) { + if ( !srcExhausted ) { + srcExhausted = !itSrc.hasNext(); + } + if ( !srcExhausted ) { + try { + src=itSrc.next(); + logger.trace("src read : " + src); + } catch (Exception e) { + logger.error("Read failure detected on " + srcReader.getDataSourceName() + ". Aborting.", e); + // Escape from the while loop + abort=true; continue; + } + } + } + + // Look-ahead dstReader if previous has been "poped" (or if never read yet) + if ( dst == null ) { + if ( !dstExhausted ) { + dstExhausted = !itDst.hasNext(); + } + if ( !dstExhausted ) { + try { + dst = itDst.next(); + logger.trace("dst read : " + dst); + } catch (NoSuchElementException e) { + logger.error("Read failure detected on " + dstReader.getDataSourceName() + ". Aborting.", e); + // Escape from the while loop + abort=true; continue; + } + } + } + + // Error-free cases (no problems while reading data) + int compare; + if ( !srcExhausted && !dstExhausted ) { + // General case : check order precedence to take an action + compare = src.compareTo(dst); + } else if ( !srcExhausted && dstExhausted ) { + // Particular case : dst is exhausted, it's like ( src < dst ) + compare=-1; + } else if ( srcExhausted && !dstExhausted ) { + // Particular case : src is exhausted, it's like ( src > dst ) + compare=1; + } else /* ( srcExhausted && dstExhausted ) */ { + // Particular case : everything is synchronized + // Exit gracefully the while loop + done=true; continue; + } + + logger.trace("compare : " + compare); + + boolean actionRealized = false; + // Take an action (insert/update/delete) + if ( compare < 0 ) { + actionRealized = _insert(src); + src = null; + // preserve dst until src key is not greater + } else if ( compare > 0 ) { + // dst current entry doesn't exists anymore (src key is greater than dst key) + actionRealized = _delete(dst); + // preserve src until dst key is not greater + dst = null; + } else /* ( compare == 0 ) */ { + // src current entry already exists in dst, update it if necessary + Set<String> changedAttr = src.getChangedAttributes(dst); + if ( ! changedAttr.isEmpty() ) { + actionRealized = _update(src,dst,changedAttr); + } else { + // Already up-to-date, nothing to do + actionRealized = true; + } + // Both src and dst have been used + src = null; + dst = null; + } + abort = !actionRealized; + } /* while */ + + return !abort; + } /* _taskRunSync() */ + + private boolean _insert(MVDataEntry entry) { + + if ( maxInserts > 0 && curInserts >= maxInserts ) { + logger.error("Max insert limit reached (" + maxInserts + ")" ); + return false; + } + + logger.debug("dstWriter : Action\n-> Insert " + entry); + try { + dstWriter.insert(entry); + } catch (Exception e) { + logger.error("Exception occured while inserting", e); + return false; + } + + curInserts++; + return true; + } + + private boolean _update(MVDataEntry updatedEntry, MVDataEntry originalEntry, Set<String> attrToUpdate) { + if ( maxUpdates > 0 && curUpdates >= maxUpdates ) { + logger.error("Max update limit reached (" + maxUpdates + ")"); + return false; + } + + logger.debug("dstWriter : Action\n-> Update " + updatedEntry + "\n-> changed attributes : " + attrToUpdate); + try { + dstWriter.update(updatedEntry, originalEntry, attrToUpdate); + } catch (Exception e) { + logger.error("Exception occured while updating", e); + return false; + } + + curUpdates++; + return true; + } + + private boolean _delete(MVDataEntry entry) { + if ( skipEntryDelete ) { + logger.info("dstWriter : skipping deletion for key " + entry.getKey()); + return true; + } + + if ( maxDeletes > 0 && curDeletes >= maxDeletes ) { + logger.error("Max delete limit reached (" + maxDeletes + ")"); + return false; + } + logger.debug("dstWriter : Action\n-> Delete " + entry); + try { + dstWriter.delete(entry); + } catch (Exception e) { + logger.error("Exception occured while deleting", e); + return false; + } + + curDeletes++; + return true; + } + + // Boring accessors + + /** + * Setter to fix limits about operations counts (safeguard) + * @param maxInserts + * @param maxUpdates + * @param maxDeletes + */ + public void setOperationLimits(int maxInserts, int maxUpdates, int maxDeletes) { + this.maxInserts = maxInserts; + this.maxUpdates = maxUpdates; + this.maxDeletes = maxDeletes; + } + + /** + * @return the skipEntryDelete + */ + public boolean isSkipEntryDelete() { + return skipEntryDelete; + } + + /** + * @param skipEntryDelete the skipEntryDelete to set + */ + public void setSkipEntryDelete(boolean skipEntryDelete) { + this.skipEntryDelete = skipEntryDelete; + } + +} diff --git a/src/main/src/utils/JVMStatsDumper.java b/src/main/src/utils/JVMStatsDumper.java new file mode 100644 index 0000000..41f1d97 --- /dev/null +++ b/src/main/src/utils/JVMStatsDumper.java @@ -0,0 +1,111 @@ +/* + * SSSync, a Simple and Stupid Synchronizer for data with multi-valued attributes + * Copyright (C) 2014 Ludovic Pouzenc <ludovic@pouzenc.fr> + * + * This file is part of SSSync. + * + * SSSync is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * SSSync is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with SSSync. If not, see <http://www.gnu.org/licenses/> + */ + +package utils; + +import java.lang.management.GarbageCollectorMXBean; +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryPoolMXBean; +import java.lang.management.MemoryUsage; +import java.lang.management.RuntimeMXBean; + +import org.apache.log4j.Level; +import org.apache.log4j.Logger; + +/** + * TODO javadoc + * + * @author lpouzenc + */ +public class JVMStatsDumper { + private static final Logger logger = Logger.getLogger(JVMStatsDumper.class.getName()); + + public static void logGCStats() { + // Skip all string construction if will not print this stuff + if ( logger.getLevel().isGreaterOrEqual(Level.INFO) ) { return; } + + long totalGarbageCollections = 0; + long garbageCollectionTime = 0; + + final String gcDumpHeader="Dumping Garbage Collector statistics\n" + + "+--------------------+-----------------------------+\n" + + "+ GC Name + Count + Time (ms) +\n" + + "+--------------------+--------------+--------------+\n"; + + StringBuilder sb = new StringBuilder(1024); + sb.append(gcDumpHeader); + + for(GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) { + + long count = gc.getCollectionCount(); + long time = gc.getCollectionTime(); + + sb.append(String.format("+ %18s + %,12d + %,12d +%n", gc.getName(), count, time)); + + if(count >= 0) totalGarbageCollections += count; + if(time >= 0) garbageCollectionTime += time; + } + + sb.append("+ + + +\n"); + sb.append(String.format("+ %18s + %,12d + %,12d +%n", + "Total", totalGarbageCollections, garbageCollectionTime + )); + sb.append("+--------------------+--------------+--------------+\n"); + + sb.append("JVM arguments : "); + RuntimeMXBean runtimeMxBean = ManagementFactory.getRuntimeMXBean(); + for ( String arg : runtimeMxBean.getInputArguments() ) { + sb.append(arg + " "); + } + + logger.debug(sb); + } + + /** + * Helper function to log the current memory usage + */ + public static void logMemoryUsage() { + // Skip all string construction if will not print this stuff + if ( logger.getLevel().isGreaterOrEqual(Level.INFO) ) { return; } + + final String memDumpHeader="Dumping memory statistics\n" + + "+--------------------------------------------------------------------------------+\n" + + "+ + Current (kio) + Peak (kio) +\n" + + "+ Pool +-----------------------------------------------------------+\n" + + "+ + Used + Committed + Used + Committed +\n" + + "+--------------------+--------------+--------------+--------------+--------------+\n"; + + StringBuilder sb = new StringBuilder(1024); + sb.append(memDumpHeader); + + for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) { + MemoryUsage peak = pool.getPeakUsage(); + MemoryUsage curr = pool.getUsage(); + sb.append(String.format("+ %18s + %,12d + %,12d + %,12d + %,12d +%n", + pool.getName(),curr.getUsed()/1024, curr.getCommitted()/1024, peak.getUsed()/1024, peak.getCommitted()/1024 + )); + pool.resetPeakUsage(); //XXX Maybe this is not a global action and is useless on a temporary object used once + } + sb.append("+--------------------+--------------+--------------+--------------+--------------+\n"); + + logger.debug(sb); + } + +} diff --git a/src/main/sssync.sh b/src/main/sssync.sh new file mode 100755 index 0000000..43a1810 --- /dev/null +++ b/src/main/sssync.sh @@ -0,0 +1,3 @@ +#!/bin/sh +cd $(dirname $0) +java -jar SSSync.jar diff --git a/src/sloc.sh b/src/sloc.sh new file mode 100755 index 0000000..6b5cb02 --- /dev/null +++ b/src/sloc.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +mkdir /tmp/sloc/{,src,JUTests} +find */src -name *.java \! -name *Bean.java -print0 | xargs -0r cp -vt /tmp/sloc/src +find */JUTests -name *.java \! -name *Bean.java -print0 | xargs -0r cp -vt /tmp/sloc/JUTests +sloccount /tmp/sloc/* +rm -r /tmp/sloc |