1 # Parameter Framework (PF) test plan 2 3 ## Notes 4 - The unit test code MAY be able to list implemented scenarios 5 in this same BDD format in order to check scenario coverage. 6 - All scenarios implicitly start by _"GIVEN a parameter framework"_. 7 - If a scenario does not specify a config files, 8 the following minimal files SHOULD be used: 9 10 Top level file (replace `{structurePath}` by its value): 11 ~~~ xml 12 <?xml version='1.0' encoding='UTF-8'?> 13 <ParameterFrameworkConfiguration SystemClassName='test' 14 TuningAllowed='false' ServerPort='1'> 15 <SubsystemPlugins/> 16 <StructureDescriptionFileLocation Path='{structurePath}'/> 17 </ParameterFrameworkConfiguration> 18 ~~~ 19 20 Structure file: 21 ~~~ xml 22 <?xml version='1.0' encoding='UTF-8'?> 23 <SystemClass Name='test'> 24 <Subsystem Name='test' Type='Virtual'> 25 <ComponentLibrary/> 26 <InstanceDefinition> 27 <BooleanParameter Name="placeholder"/> 28 </InstanceDefinition> 29 </Subsystem> 30 </SystemClass> 31 ~~~ 32 33 ## Starting the Parameter Framework 34 35 - set a logger 36 - [X] Scenario: Default logger 37 * WHEN no logger is set 38 * THEN start should succeed 39 40 - [X] Scenario: No logger 41 * WHEN a nullptr logger is set 42 * THEN start should succeed 43 44 - [X] Scenario: Logger should receive info and warnings 45 * GIVEN config files that emit warnings 46 * GIVEN a logger that store logs 47 * WHEN the record logger is set 48 * THEN start should succeed 49 * AND_THEN the logger should have stored info and warning log 50 51 - [X] Scenario: Unset logger 52 * GIVEN config files that emit warnings 53 * GIVEN a logger that store logs 54 * WHEN the record logger is set 55 * AND_WHEN a nullptr logger is set 56 * THEN start should succeed 57 * AND_THEN the record logger should NOT have stored any info or warning log 58 59 - set/get various properties before startup: 60 - "force no remote interface" 61 - [ ] Scenario: Tuning OK 62 * GIVEN config files with tuning enabled on a valid port 63 * WHEN the remote interface is NOT forbiden 64 * THEN start should succeed 65 66 - [ ] Scenario: Invalid tunning 67 * GIVEN config files with tuning enabled on an invalid port (0?) 68 * WHEN the remote interface is NOT forbiden 69 * THEN start should fail 70 71 - [ ] Scenario: Forbiden invalid tunning 72 * GIVEN config files with tuning enabled on an invalid port (0?) 73 * WHEN the remote interface is forbiden 74 * THEN start should succeed 75 76 - "failure on missing subsystem" 77 - [ ] Scenario: Plugin OK 78 * GIVEN config files with a valid plugin 79 * WHEN the missing subsystem policy is set to failure 80 * THEN start should succeed 81 82 - [X] Scenario: Wrong plugin 83 * GIVEN config files with a invalid plugin 84 * WHEN the missing subsystem policy is set to failure 85 * THEN start should fail 86 87 - [ ] Scenario: Wrong plugin but fallback 88 * GIVEN config files with a invalid plugin 89 * WHEN the missing subsystem policy is set to success 90 * THEN start should succeed 91 92 - "failure on failed settings load" 93 - [X] Scenario: Settings OK 94 * GIVEN config files with a valid settings file 95 * WHEN settings load failure is requested to abort start 96 * THEN start should succeed 97 98 - [X] Scenario: Wrong settings 99 * GIVEN config files with a invalid (non existant?) settings file 100 * WHEN settings load failure is requested to abort start 101 * THEN start should fail 102 103 - [X] Scenario: Wrong settings but ignore 104 * GIVEN config files with a invalid (non existant?) settings file 105 * WHEN settings load failure is requested to abort start 106 * THEN start should succeed 107 108 - "schema folder location" and "validate schema on start" 109 - [ ] Scenario: Schema OK 110 * GIVEN config files with correct default schema location 111 * WHEN schema folder location is left to default 112 * WHEN schema validation is enabled 113 * THEN start should succeed 114 115 - [ ] Scenario: Inexisting schemas 116 * GIVEN config files with correct default schema location 117 * WHEN schema folder location is set to an invalid location (/doesNotExist ?) 118 * WHEN schema validation is enabled 119 * THEN start should fail 120 121 - [ ] Scenario: Inexisting ignored schemas 122 * GIVEN config files with correct default schema location 123 * WHEN schema folder location is set to an invalid location (/doesNotExist ?) 124 * WHEN schema validation is disabled 125 * THEN start should succeed 126 127 - [ ] Scenario: Inexisting used schemas 128 * GIVEN config files with custom schema location (./shemaCustomFolderName) 129 * WHEN schema folder location is set to the custom shema location 130 * WHEN schema validation is enabled 131 * THEN start should succeed 132 133 - [ ] Scenario: Schema OK, config KO 134 * GIVEN incorect config files with correct default schema location 135 * WHEN schema validation is enabled 136 * THEN start should fail 137 138 ### Error cases/special cases 139 140 - invalid configuration or a file can't be read: 141 - [X] Scenario: invalid top level config file 142 * GIVEN an incorect (empty ? wrong balisa ?) config top level file. 143 * THEN start should fail 144 145 - [X] Scenario: invalid structure 146 * GIVEN an incorect (empty ? wrong balisa ?) structure file. 147 * THEN start should fail 148 149 - [X] Scenario: invalid settings 150 * GIVEN an incorect (empty ? wrong balisa ?) settings file. 151 * THEN start should fail 152 153 - plugins can't be found (unless silenced by the user) 154 - Done in "failure on missing subsystem" 155 156 - usage of `<xi:include .../>` 157 - [ ] Scenario: Arbitrary xincluded node 158 * GIVEN a top level file xincluding a file with the "SettingsConfiguration" balisa 159 * THEN start should succeed 160 161 - plugins: 162 - non-empty "Location" attribute 163 - [ ] Scenario: Plugin location OK 164 * GIVEN a top level file with a plugin folder location attribute 165 * THEN start should succeed 166 - empty "Location" attribute 167 - relative path 168 - [ ] Scenario: Empty Location, standard plugin install 169 * GIVEN config files with an empty location attribute 170 * GIVEN a plugin installed in a dlopen standard path 171 * GIVEN that "Location/Plugin[name]" is the plugin name 172 * THEN start should succeed 173 174 - [ ] Scenario: Empty Location, non standard plugin install 175 * GIVEN config files with an empty location attribute 176 * GIVEN a plugin NOT installed in a dlopen standard path 177 * GIVEN that "Location/Plugin[name]" is the plugin name 178 * THEN start should fail 179 180 - absolute path 181 - [ ] Scenario: Empty Location, plugin absolute path 182 * GIVEN config files with an empty location attribute 183 * GIVEN a plugin 184 * GIVEN that "Location/Plugin[name]" is the plugin absolute path 185 * THEN start should succeed 186 187 - [ ] Scenario: Empty Location, plugin non existing path 188 * GIVEN config files with an empty location attribute 189 * GIVEN that "Location/Plugin[name]" is an non existing path (/libDoesNotExist.so?) 190 * THEN start should fail 191 192 * * * 193 194 ## Logging 195 196 - info and warning 197 - Partly covered by the "set a logger" section 198 - [ ] Scenario: Plugin info and warning log 199 * GIVEN a plugin that log a unique string as info and warning 200 * GIVEN a logger that store logs 201 * WHEN the record logger is set 202 * THEN start should succeed 203 * AND_THEN the logger should have stored the plugin specific info and warning log 204 205 * * * 206 207 ## Basic Domains & Configurations management 208 209 - [ ] Scenario: create and delete an empty domain 210 * GIVEN a started parameter framework without domains 211 * THEN creating a domain with an arbitrary name ("domain1"?) should succeed 212 * (AND_THEN the domain list should contain the domain name) 213 * THEN deleting the domain should succeed 214 215 - [ ] Scenario: create and delete a configuration 216 * GIVEN a started parameter framework with a domain 217 * THEN creating a configuration in this domain with an arbitrary name ("config1") should succeed 218 * (AND_THEN the domain configuration list should contain the configuration name) 219 * THEN deleting the configuration should succeed 220 * (AND_THEN the domain configuration list should not contain the configuration name) 221 222 - [ ] Scenario: delete a domain with configurations 223 * GIVEN a started parameter framework with a domain and several configuration (3?) 224 * THEN deleting the domain should succeed 225 * (AND_THEN the domain list should not contain the domain name) 226 227 - [ ] Scenario: delete all domains 228 * GIVEN a started parameter framework with several domains 229 * THEN deleting all domains should succeed 230 * (AND_THEN the domain list should be empty) 231 232 - [ ] Scenario: rename a domain 233 * GIVEN a started parameter framework with a domain and several configuration (3?) 234 * THEN renaming the domain to an arbitrary name should succeed 235 236 237 ### Error cases/special cases 238 239 - [ ] Scenario: create a domain with an already-existing name 240 * GIVEN a started parameter framework with a domain (arbitrary name) 241 * THEN creating a second domain with the same name than the first should fail 242 243 - [ ] Scenario: create a configuration with an already-existing name 244 * GIVEN a started parameter framework with a domain 245 containing one configuration (arbitrary name) 246 * THEN creating a second configuration in this domain 247 with the same name than the fist should fail 248 249 - [ ] delete a non-existent domain 250 * GIVEN a started parameter framework without domain 251 * THEN deleting an arbitrary named domain should fail 252 253 - [ ] delete a non-existent configuration 254 * GIVEN a started parameter framework one domain 255 * THEN deleting in this domain a arbitrary named configuration should fail 256 257 - [ ] list configuration of a unknown domain 258 * GIVEN a started parameter framework without domain 259 * THEN listing configuration of an arbitrary named domain should fail 260 261 - [ ] Scenario: rename a domain to an already used name 262 * GIVEN a started parameter framework with two domains 263 with arbitrary different names ("domain1" and "domain2") 264 * THEN renaming first domain ("domain1") to the name of the second one ("domain2") should fail 265 266 - [ ] Scenario: rename a configuration to an already used name 267 * GIVEN a started parameter framework with one domain containing two configuration 268 with arbitrary different names ("config1" and "config2") 269 * THEN renaming first configuration ("config1") 270 to the name of the second one ("config2") should fail 271 272 - [ ] Scenario: rename a domain to its own name 273 * GIVEN a started parameter framework one arbitrary named domain 274 * THEN renaming the domain to its own name should succeed 275 276 - [ ] rename a configuration to its own name 277 * GIVEN a started parameter framework a domain and one configuration 278 * THEN renaming the configuration to its own name should succeed 279 280 * * * 281 282 ## Domain modification 283 284 - set/get sequence awareness 285 - add/list/remove elements to a domain 286 - *(split domain ?)* 287 - *(listBelongingDomains ?)* 288 - *(listAssociatedDomains ?)* 289 - *(listAssociatedElements ?)* 290 - *(listConflictingElements ?)* 291 - *(listRogueElements ?)* 292 293 ### Error cases/special cases 294 295 - add (or remove) a non-existent element 296 - add (or remove) a parameter to (or from) a non-existent domain 297 298 * * * 299 300 ## Configurations modification 301 302 - set a configuration rule; get it back 303 - trivial rules, complex rules (fuzzing ?) 304 - clear a rule 305 - save/restore a configuration 306 - set/get element sequence 307 - set/get a parameter for a given configuration 308 - set a parameter belonging to a domain and save a configuration 309 310 ### Error cases 311 312 - set a wrongly-formatted rule 313 - set a rule using an unknown criterion (or unknown criterion value) 314 - set/get a rule to a non-existent domain (or configuration) 315 316 * * * 317 318 ## Criteria 319 320 - create a criterion 321 - exclusive 322 - inclusive 323 - *(list criteria)* 324 - set a criterion; get a criterion value 325 - exclusive criterion 326 - inclusive criterion 327 - special case: "none" value 328 329 ### Error cases/special cases 330 331 - duplicated criterion name (with the same type/with a different type) 332 - duplicated criterion value 333 - max supported number of criterion values 334 335 * * * 336 337 ## Settings Import/Export 338 339 - import xml domains 340 - w/ or w/o settings 341 - from a file/from a string 342 - all/single 343 - export xml domains 344 - w/ or w/o settings 345 - to a file/from a string 346 - all/single 347 348 ### Error case/special cases 349 350 - invalid settings 351 - invalid xml (should we use a fuzzing framework ?) 352 - valid xml but not matching the XSD (ditto) 353 - valid but unusable data: 354 - unknown criterion/criterion values 355 - unknown parameters 356 - unreadable/unwritable file 357 - single domain import overwrite 358 - fail if overwrite not requested 359 - succeed if overwrite requested 360 361 * * * 362 363 ## Parameters 364 365 - structures are correctly instantiated 366 - parameters in the "instance definition" 367 - component types instantiation 368 - parameter blocks 369 - bit blocks 370 - arrays 371 - check parameter mappings are correctly applied 372 - at all levels possible (subsystem, component type, component instance, 373 block, parameter...) 374 - context mappings, amends, instantiation mappings 375 - set/get a parameter 376 - parameter handles 377 - test on parameter values 378 - booleans 379 - bit blocks 380 - numeric types ranges 381 - fixed-point tests 382 - string length 383 384 * * * 385 386 ## Configuration application 387 388 - apply configurations 389 - sequence aware domains 390 - forced sync 391 392 * * * 393 394 ## Misc/to be classified 395 396 - tuning mode 397 - auto sync 398 - remote interface instantiation 399 - value space (raw/real)/format (dec/hex) 400 401 ### Error cases 402 403
