1 """
2 This is a top level package, hosting the entire CSB test framework. It is divided
3 into several major parts:
4
5 - test cases, located under csb.test.cases
6 - test data, in C{/csb/test/data} (not a package)
7 - test console, in C{/csb/test/app.py}
8
9 This module, csb.test, contains all the glue-code functions, classes and
10 decorators you would need in order to write tests for CSB.
11
12 1. Configuration and Tree
13
14 L{Config<csb.test.Config>} is a common config object shared between CSB
15 tests. Each config instance contains properties like:
16
17 - data: the data folder, automatically discovered and loaded in
18 csb.test.Config.DATA at module import time
19 - temp: a default temp folder, which test cases can use
20
21 Each L{Config<csb.test.Config>} provides a convenient way to retrieve
22 files from C{/csb/test/data}. Be sure to check out L{Config.getTestFile}
23 and L{Config.getPickle}. In case you need a temp file, use
24 L{Config.getTempStream} or have a look at L{csb.io.TempFile} and
25 L{csb.io.TempFolder}.
26
27 All test data files should be placed in the C{data} folder. All test
28 modules must be placed in the root package: csb.test.cases. There is
29 a strict naming convention for test modules: the name of a test module
30 should be the same as the name of the CSB API package it tests. For
31 example, if you are writing tests for C{csb/bio/io/__init__.py}, the
32 test module must be C{csb/test/cases/bio/io/__init__.py}. C{csb.test.cases}
33 is the root package of all test modules in CSB.
34
35 2. Writing Tests
36
37 Writing a test is easy. All you need is to import csb.test and then
38 create your own test cases, derived from L{csb.test.Case}:
39
40 >>> import csb.test
41 >>> @csb.test.unit
42 class TestSomeClass(csb.test.Case):
43 def setUp(self):
44 super(TestSomeClass, self).setUp()
45 # do something with self.config here...
46
47 In this way your test case instance is automatically equipped with a
48 reference to the test config, so your test method can be:
49
50 >>> @csb.test.unit
51 class TestSomeClass(csb.test.Case):
52 def testSomeMethod(self):
53 myDataFile = self.config.getTestFile('some.file')
54 self.assert...
55
56 The "unit" decorator marks a test case as a collection of unit tests.
57 All possibilities are: L{csb.test.unit}, L{csb.test.functional}, L{csb.test.custom},
58 and L{csb.test.regression}.
59
60 Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit
61 more work. Custom tests must be functions, not classes. Basically a
62 custom test is a function, which builds a unittest.TestSuite instance
63 and then returns it when called without arguments.
64
65 Regression tests are usually created in response to reported bugs. Therefore,
66 the best practice is to mark each test method with its relevant bug ID:
67
68 >>> @csb.test.regression
69 class SomeClassRegressions(csb.test.Case)
70 def testSomeFeature(self)
71 \"""
72 @see: [CSB 000XXXX]
73 \"""
74 # regression test body...
75
76 3. Style Guide:
77
78 - name test case packages as already described
79 - group tests in csb.test.Case-s and name them properly
80 - prefix test methods with "test", like "testParser" - very important
81 - use camelCase for methods and variables. This applies to all the
82 code under csb.test (including test) and does not apply to the rest
83 of the library!
84 - for functional tests it's okay to define just one test method: runTest
85 - for unit tests you should create more specific test names, for example:
86 "testParseFile" - a unit test for some method called "parse_file"
87 - use csb.test decorators to mark tests as unit, functional, regression, etc.
88 - make every test module executable::
89
90 if __name__ == '__main__':
91 csb.test.Console() # Discovers and runs all test cases in the module
92
93 4. Test Execution
94
95 Test discovery is handled by C{test builders} and a test runner
96 C{app}. Test builders are subclasses of L{AbstractTestBuilder}.
97 For every test type (unit, functional, regression, custom) there is a
98 corresponding test builder. L{AnyTestBuilder} is a special builder which
99 scans for unit, regression and functional tests at the same time.
100
101 Test builder classes inherit the following test discovery methods:
102
103 - C{loadTests} - load tests from a test namespace. Wildcard
104 namespaces are handled by C{loadAllTests}
105 - C{loadAllTests} - load tests from the given namespace, and
106 from all sub-packages (recursive)
107 - C{loadFromFile} - load tests from an absolute file name
108 - C{loadMultipleTests} - calls C{loadTests} for a list of
109 namespaces and combines all loaded tests in a single suite
110
111 Each of those return test suite objects, which can be directly executed
112 with python's unittest runner.
113
114 Much simpler way to execute a test suite is to use our test app
115 (C{csb/test/app.py}), which is simply an instance of L{csb.test.Console}::
116
117 $ python csb/test/app.py --help
118
119 The app has two main arguments:
120
121 - test type - tells the app which TestBuilder to use for test dicsovery
122 ("any" triggers L{AnyTestBuilder}, "unit" - L{UnitTestBuilder}, etc.)
123 - test namespaces - a list of "dotted" test modules, for example::
124
125 csb.test.cases.bio.io.* # io and sub-packages
126 csb.test.cases.bio.utils # only utils
127 . # current module
128
129 In addition to running the app from the command line, you can run it
130 also programmatically by instantiating L{csb.test.Console}. You can
131 construct a test console object by passing a list of test namespace(s)
132 and a test builder class to the Console's constructor.
133
134
135 5. Commit Policies
136
137 Follow these guidelines when making changes to the repository:
138
139 - B{no bugs in "trunk"}: after fixing a bug or implementing a new
140 feature, make sure at least the default test set passes by running
141 the test console without any arguments. This is equivalent to:
142 app.py -t any "csb.test.cases.*". (If no test case from this set covers
143 the affected code, create a test case first, as described in the other
144 policies)
145
146 - B{no recurrent issues}: when a bug is found, first write a regression
147 test with a proper "@see: BugID" tag in the docstring. Run the test
148 to make sure it fails. After fixing the bug, run the test again before
149 you commit, as required by the previous policy
150
151 - B{test all new features}: there should be a test case for every new feature
152 we implement. One possible approach is to write a test case first and
153 make sure it fails; when the new feature is ready, run the test again
154 to make sure it passes
155
156 @warning: for compatibility reasons do NOT import and use the unittest module
157 directly. Always import unittest from csb.test, which is guaranteed
158 to be python 2.7+ compatible.
159 """
160 import os
161 import sys
162 import imp
163 import types
164 import time
165 import tempfile
166 import traceback
167 import argparse
168
169 import csb.io
170 import csb.core
171
172 try:
173 from unittest import skip, skipIf
174 import unittest
175 except ImportError:
176 import unittest2 as unittest
177
178 from abc import ABCMeta, abstractproperty
187
189 """
190 General CSB Test Config. Config instances contain the following properties:
191
192 - data - path to the CSB Test Data directory. Default is L{Config.DATA}
193 - temp - path to the system's temp directory. Default is L{Config.TEMP}
194 - config - the L{Config} class
195 """
196
197 DATA = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data')
198 """
199 @cvar: path to the default test data directory: <install dir>/csb/test/data
200 """
201 GENERATED_DATA = DATA
202 """
203 @cvar: path to the default data directory for generated test files
204 """
205 TEMP = os.path.abspath(tempfile.gettempdir())
206 """
207 @cvar: path to the default system's temp directory
208 """
209
210 @staticmethod
212 """
213 Override the default L{Config.DATA} with a new data root directory.
214
215 @param path: full directory path
216 @type path: str
217 """
218 if not os.path.isdir(path):
219 raise IOError('Path not found: {0}'.format(path))
220
221 Config.DATA = os.path.abspath(path)
222
223 @staticmethod
225 """
226 Override the default L{Config.GENERATED_DATA} with a new data root directory.
227
228 @param path: full directory path
229 @type path: str
230 """
231 if not os.path.isdir(path):
232 raise IOError('Path not found: {0}'.format(path))
233
234 Config.GENERATED_DATA = os.path.abspath(path)
235
236 @property
238 """
239 Test data directory
240 @rtype: str
241 """
242 return Config.DATA
243
244 @property
246 """
247 Test data directory for generated files
248 @rtype: str
249 """
250 return Config.GENERATED_DATA
251
252 @property
254 """
255 Test temp directory
256 @rtype: str
257 """
258 return Config.TEMP
259
261 """
262 Search for C{fileName} in the L{Config.DATA} directory. If not found,
263 try also L{Config.GENERATED_DATA} (if different).
264
265 @param fileName: the name of a test file to retrieve
266 @type fileName: str
267 @param subDir: scan a sub-directory of L{Config.DATA}
268 @type subDir: str
269
270 @return: full path to C{fileName}
271 @rtype: str
272
273 @raise IOError: if no such file is found
274 """
275 for data in [self.data, self.generatedData]:
276 file = os.path.join(data, subDir, fileName)
277
278 if os.path.isfile(file):
279 return file
280
281 raise IOError('Test file not found: {0}'.format(fileName))
282
284 """
285 Same as C{self.getTestFile}, but try to unpickle the the file
286 and return the unpickled object. Pickles are usually stored in
287 L{Config.GENERATED_DATA}.
288
289 @param fileName: the name of a test file to retrieve
290 @type fileName: str
291 @param subDir: scan a sub-directory of L{Config.DATA}
292 @type subDir: str
293
294 @rtype: object
295 """
296 file = self.getTestFile(fileName, subDir)
297 return csb.io.Pickle.load(open(file, 'rb'))
298
299 - def getContent(self, fileName, subDir=''):
300 """
301 Same as C{self.getTestFile}, but also read and return the contents of
302 the file.
303
304 @param fileName: the name of a test file to retrieve
305 @type fileName: str
306 @param subDir: scan a sub-directory of L{Config.DATA}
307 @type subDir: str
308
309 @rtype: str
310 """
311 with open(self.getTestFile(fileName, subDir)) as f:
312 return f.read()
313
315 """
316 Return a temporary file stream::
317
318 with self.getTempStream() as tmp:
319 tmp.write(something)
320 tmp.flush()
321 file_name = tmp.name
322
323 @param mode: file open mode (text, binary), default=t
324 @type mode: str
325 @rtype: file stream
326 """
327 return csb.io.TempFile(mode=mode)
328
330 """
331 Try to deserialize some pickled data files. Call L{Config.updateDataFiles}
332 if the pickles appeared incompatible with the current interpreter.
333 """
334 try:
335 self.getPickle('1nz9.model1.pickle')
336 except:
337 self.updateDataFiles()
338
362
363 -class Case(unittest.TestCase):
364 """
365 Base class, defining a CSB Test Case. Provides a default implementation
366 of C{unittest.TestCase.setUp} which grabs a reference to a L{Config}.
367 """
368
369 @property
371 """
372 Test config instance
373 @rtype: L{Config}
374 """
375 return self.__config
376
378 """
379 Provide a reference to the CSB Test Config in the C{self.config} property.
380 """
381 self.__config = Config()
382 assert hasattr(self.config, 'data'), 'The CSB Test Config must contain the data directory'
383 assert self.config.data, 'The CSB Test Config must contain the data directory'
384
386 """
387 Re-raise the last exception with its full traceback, but modify the
388 argument list with C{addArgs} and the original stack trace.
389
390 @param addArgs: additional arguments to append to the exception
391 @type addArgs: tuple
392 """
393 klass, ex, _tb = sys.exc_info()
394 ex.args = list(ex.args) + list(addArgs) + [''.join(traceback.format_exc())]
395
396 raise klass(ex.args)
397
399
400 if first == second:
401 return
402 if delta is not None and places is not None:
403 raise TypeError("specify delta or places not both")
404
405 if delta is not None:
406
407 if abs(first - second) <= delta:
408 return
409
410 m = '{0} != {1} within {2} delta'.format(first, second, delta)
411 msg = self._formatMessage(msg, m)
412
413 raise self.failureException(msg)
414
415 else:
416 if places is None:
417 places = 7
418
419 return super(Case, self).assertAlmostEqual(first, second, places=places, msg=msg)
420
422 """
423 Fail if it took more than C{duration} seconds to invoke C{callable}.
424
425 @param duration: maximum amount of seconds allowed
426 @type duration: float
427 """
428
429 start = time.time()
430 callable(*args, **kargs)
431 execution = time.time() - start
432
433 if execution > duration:
434 self.fail('{0}s is slower than {1}s)'.format(execution, duration))
435
436 @classmethod
438 """
439 Run this test case.
440 """
441 suite = unittest.TestLoader().loadTestsFromTestCase(cls)
442 runner = unittest.TextTestRunner()
443
444 return runner.run(suite)
445
448
450 """
451 This is a base class, defining a test loader which exposes the C{loadTests}
452 method.
453
454 Subclasses must override the C{labels} abstract property, which controls
455 what kind of test cases are loaded by the test builder.
456 """
457
458 __metaclass__ = ABCMeta
459
460 @abstractproperty
463
465 """
466 Load L{csb.test.Case}s from a module file.
467
468 @param file: test module file name
469 @type file: str
470
471 @return: a C{unittest.TestSuite} ready for the test runner
472 @rtype: C{unittest.TestSuite}
473 """
474 mod = self._loadSource(file)
475 suite = unittest.TestLoader().loadTestsFromModule(mod)
476 return unittest.TestSuite(self._filter(suite))
477
479 """
480 Load L{csb.test.Case}s from the given CSB C{namespace}. If the namespace
481 ends with a wildcard, tests from sub-packages will be loaded as well.
482 If the namespace is '__main__' or '.', tests are loaded from __main__.
483
484 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
485 load tests from '/csb/test/cases/bio/__init__.py'
486 @type namespace: str
487
488 @return: a C{unittest.TestSuite} ready for the test runner
489 @rtype: C{unittest.TestSuite}
490 """
491 if namespace.strip() == '.*':
492 namespace = '__main__.*'
493 elif namespace.strip() == '.':
494 namespace = '__main__'
495
496 if namespace.endswith('.*'):
497 return self.loadAllTests(namespace[:-2])
498 else:
499 loader = unittest.TestLoader()
500 tests = loader.loadTestsFromName(namespace)
501 return unittest.TestSuite(self._filter(tests))
502
504 """
505 Load L{csb.test.Case}s from a list of given CSB C{namespaces}.
506
507 @param namespaces: a list of test module namespaces, e.g.
508 ('csb.test.cases.bio', 'csb.test.cases.bio.io') will
509 load tests from '/csb/test/cases/bio.py' and
510 '/csb/test/cases/bio/io.py'
511 @type namespaces: tuple of str
512
513 @return: a C{unittest.TestSuite} ready for the test runner
514 @rtype: C{unittest.TestSuite}
515 """
516 if not csb.core.iterable(namespaces):
517 raise TypeError(namespaces)
518
519 return unittest.TestSuite(self.loadTests(n) for n in namespaces)
520
522 """
523 Load L{csb.test.Case}s recursively from the given CSB C{namespace} and
524 all of its sub-packages. Same as::
525
526 builder.loadTests('namespace.*')
527
528 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
529 load tests from /csb/test/cases/bio/*'
530 @type namespace: str
531
532 @return: a C{unittest.TestSuite} ready for the test runner
533 @rtype: C{unittest.TestSuite}
534 """
535 suites = []
536
537 try:
538 base = __import__(namespace, level=0, fromlist=['']).__file__
539 except ImportError:
540 raise InvalidNamespaceError('Namespapce {0} is not importable'.format(namespace))
541
542 if os.path.splitext(os.path.basename(base))[0] != '__init__':
543 suites.append(self.loadTests(namespace))
544
545 else:
546
547 for entry in os.walk(os.path.dirname(base)):
548
549 for item in entry[2]:
550 file = os.path.join(entry[0], item)
551 if extension and item.endswith(extension):
552 suites.append(self.loadFromFile(file))
553
554 return unittest.TestSuite(suites)
555
557 """
558 Import and return the Python module identified by C{path}.
559
560 @note: Module objects behave as singletons. If you import two different
561 modules and give them the same name in imp.load_source(mn), this
562 counts for a redefinition of the module originally named mn, which
563 is basically the same as reload(mn). Therefore, you need to ensure
564 that for every call to imp.load_source(mn, src.py) the mn parameter
565 is a string that uniquely identifies the source file src.py.
566 """
567 name = os.path.splitext(os.path.abspath(path))[0]
568 name = name.replace('.', '-').rstrip('__init__').strip(os.path.sep)
569
570 return imp.load_source(name, path)
571
573 """
574 Extract test cases recursively from a test C{obj} container.
575 """
576 cases = []
577 if isinstance(obj, unittest.TestSuite) or csb.core.iterable(obj):
578 for item in obj:
579 cases.extend(self._recurse(item))
580 else:
581 cases.append(obj)
582 return cases
583
585 """
586 Filter a list of objects using C{self.labels}.
587 """
588 filtered = []
589
590 for test in self._recurse(tests):
591 for label in self.labels:
592 if hasattr(test, label) and getattr(test, label) is True:
593 filtered.append(test)
594
595 return filtered
596
598 """
599 Build a test suite of cases, marked as either unit, functional or regression
600 tests. For detailed documentation see L{AbstractTestBuilder}.
601 """
602 @property
605
607 """
608 Build a test suite of cases, marked as unit tests.
609 For detailed documentation see L{AbstractTestBuilder}.
610 """
611 @property
614
616 """
617 Build a test suite of cases, marked as functional tests.
618 For detailed documentation see L{AbstractTestBuilder}.
619 """
620 @property
623
625 """
626 Build a test suite of cases, marked as regression tests.
627 For detailed documentation see L{AbstractTestBuilder}.
628 """
629 @property
632
634 """
635 Build a test suite of cases, marked as custom tests. CustomTestBuilder will
636 search for functions, marked with the 'custom' test decorator, which return
637 a dynamically built C{unittest.TestSuite} object when called without
638 parameters. This is convenient when doing data-related tests, e.g.
639 instantiating a single type of a test case many times iteratively, for
640 each entry in a database.
641
642 For detailed documentation see L{AbstractTestBuilder}.
643 """
644 @property
647
649
650 mod = self._loadSource(file)
651 suites = self._inspect(mod)
652
653 return unittest.TestSuite(suites)
654
671
673
674 objects = map(lambda n: getattr(module, n), dir(module))
675 return self._filter(objects)
676
678 """
679 Filter a list of objects using C{self.labels}.
680 """
681 filtered = []
682
683 for obj in factories:
684 for label in self.labels:
685 if hasattr(obj, label) and getattr(obj, label) is True:
686 suite = obj()
687 if not isinstance(suite, unittest.TestSuite):
688 raise ValueError('Custom test function {0} must return a '
689 'unittest.TestSuite, not {1}'.format(obj.__name__, type(suite)))
690 filtered.append(suite)
691
692 return filtered
693
695 """
696 A class decorator, used to label unit test cases.
697
698 @param klass: a C{unittest.TestCase} class type
699 @type klass: type
700 """
701 if not isinstance(klass, type):
702 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
703
704 setattr(klass, Attributes.UNIT, True)
705 return klass
706
708 """
709 A class decorator, used to label functional test cases.
710
711 @param klass: a C{unittest.TestCase} class type
712 @type klass: type
713 """
714 if not isinstance(klass, type):
715 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
716
717 setattr(klass, Attributes.FUNCTIONAL, True)
718 return klass
719
721 """
722 A class decorator, used to label regression test cases.
723
724 @param klass: a C{unittest.TestCase} class type
725 @type klass: type
726 """
727 if not isinstance(klass, type):
728 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
729
730 setattr(klass, Attributes.REGRESSION, True)
731 return klass
732
734 """
735 A function decorator, used to mark functions which build custom (dynamic)
736 test suites when called.
737
738 @param function: a callable object, which returns a dynamically compiled
739 C{unittest.TestSuite}
740 @type function: callable
741 """
742 if isinstance(function, type):
743 raise TypeError("Can't apply function decorator on a class")
744 elif not hasattr(function, '__call__'):
745 raise TypeError("Can't apply function decorator on non-callable {0}".format(type(function)))
746
747 setattr(function, Attributes.CUSTOM, True)
748 return function
749
750 -def skip(reason, condition=None):
751 """
752 Mark a test case or method for skipping.
753
754 @param reason: message
755 @type reason: str
756 @param condition: skip only if the specified condition is True
757 @type condition: bool/expression
758 """
759 if isinstance(reason, types.FunctionType):
760 raise TypeError('skip: no reason specified')
761
762 if condition is None:
763 return unittest.skip(reason)
764 else:
765 return unittest.skipIf(condition, reason)
766
768 """
769 Build and run all tests of the specified namespace and kind.
770
771 @param namespace: a dotted name, which specifies the test module
772 (see L{csb.test.AbstractTestBuilder.loadTests})
773 @type namespace: str
774 @param builder: test builder to use
775 @type builder: any L{csb.test.AbstractTestBuilder} subclass
776 @param verbosity: verbosity level for C{unittest.TestRunner}
777 @type verbosity: int
778 @param update: if True, refresh all pickles in csb/test/data
779 @type update: bool
780 @param generated_data: where to cache generated test files (directory)
781 @type generated_data: str
782 """
783
784 BUILDERS = {'unit': UnitTestBuilder, 'functional': FunctionalTestBuilder,
785 'custom': CustomTestBuilder, 'any': AnyTestBuilder,
786 'regression': RegressionTestBuilder}
787
788
810
811 @property
813 return self._namespace
814 @namespace.setter
816 if csb.core.iterable(value):
817 self._namespace = list(value)
818 else:
819 self._namespace = [value]
820
821 @property
824 @builder.setter
826 self._builder = value
827
828 @property
830 return self._verbosity
831 @verbosity.setter
833 self._verbosity = value
834
835 @property
838
839 @property
842
843 @property
846 @update.setter
848 self._update = bool(value)
849
850 @property
853 @generated_data.setter
856
871
873
874 parser = argparse.ArgumentParser(prog=self.program, description="CSB Test Runner Console.")
875
876 parser.add_argument("-t", "--type", type=str, default="any", choices=list(Console.BUILDERS),
877 help="Type of tests to load from each namespace (default=any)")
878 parser.add_argument("-v", "--verbosity", type=int, default=1,
879 help="Verbosity level passed to unittest.TextTestRunner (default=1).")
880 parser.add_argument("-u", "--update-files", default=False, action="store_true",
881 help="Force update of the test pickles in " + Config.GENERATED_DATA)
882 parser.add_argument("-g", "--generated-resources", type=str, default=Config.GENERATED_DATA,
883 help="Generate, store and load additional test resources in this directory"
884 " (default=" + Config.GENERATED_DATA + ")")
885
886 parser.add_argument("namespaces", nargs='*',
887 help="""An optional list of CSB test dotted namespaces, from which to
888 load tests. '__main__' and '.' are interpreted as the
889 current module. If a namespace ends with an asterisk
890 '.*', all sub-packages will be scanned as well.
891
892 Examples:
893 "csb.test.cases.bio.*"
894 "csb.test.cases.bio.io" "csb.test.cases.bio.utils"
895 ".")""")
896
897 args = parser.parse_args(argv)
898
899 self.builder = Console.BUILDERS[args.type]
900 self.verbosity = args.verbosity
901 self.update = args.update_files
902 self.generated_data = args.generated_resources
903
904 if args.namespaces:
905 self.namespace = args.namespaces
906
907
908 if __name__ == '__main__':
909
910 Console()
911