debugging

Debugging

• InfoFiles
• Blocking the download of the datafiles
• Testing the datafiles
• Testing the sourcefiles and their output
• Coming in the near future
• Coming in the further future

InfoFiles

tv_grab_py_APY has a class to collect debugging information for the develloper called InfoFiles. Its functionality is activated by adding to your configuration file:

write_info_files = True

It creates some extra output and can mail you some data. To get mail you have to set up the log_file mailing, adapting the mail_log_address, mailserver and mailport , and add the extra mail_info_address. If you do not define this address mail_log_address is used:

mail_log = True
mail_log_address = postmaster
mailserver = localhost
mailport = 25
mail_info_address = <your mailaddress>

The extra files created on every run in ~/.xmltv/ are (the 3 was added to the names to distinguish from the old tv_grab_nl_py InfoFiles):

• fetched-programs3:
  A per channel and source overview of collected programmes and the on every merge resulting list. Any item recognized as a groupslot or a gap is preceded by a #. Any item retrieved from the cash is preceded by a *.
• xml_output3:
  The normal XMLTV output.
• detail_output3:
  Cumulative data on several data collecting and manipulation processes, including funcid 201 data
• raw_output3:
  Used by me to collect raw data on deep debugging. Should normally always be empty.

The mail-messages are, replacing Tv_grab_nl_py with the name of your grabber as set in your frontend:

• Tv_grab_nl_py lineup changes:
  A list of lineup changes consisting of:
  • New channelids reported by the source, but not defined in source_channels
  • Removed channelids defined in source_channels but not reported by the source
  • Empty channelids mentioned in empty_channels but no longer reported by the source.
• Tv_grab_nl_py url failures:
  A list of failed urls with an error added, these are only URL, HTTP and time-out failures.

Blocking the download of the datafiles

If you add:

self.only_local_sourcefiles = True

to your frontend after the line:

tvgrabpyAPI.Configure.__init__(self)

Downloading the datafiles will not be attempted. Only locally in ~/.xmltv/sources and in /var/lib/tvgrabpyAPI will be looked.

Testing the datafiles

Since release beta-1.0.2-p20160820 there is a script: tv_grab_test_json.py in the package that can verify either a grabber_datafile or a sourcefile. It takes the name, including the path to the file, as an option and uses several data-files saved to ~\.json_struct it retrieves on startup together with tv_grab_API.json. It uses functionality from both this API and from DataTreeGrab 1.2 and does the following:

• It tries to load the json file identifying it as a grabber_datafile, a sourcefile (virtual, basic or with detail pages) or an unknown json-file. On failure the output from the json module is shown, which most of the time gives you a line and column number.
• It verifies the presence of possible root-keys (dividing them into "required", "sugested" and "optional") and checks as far as possibly on the used syntax and reports to the stdout. Output to a file is forthcoming, but you can simply pipe the output to a file. Most syntactical errors and options are found, but do report on unclear or wrong output.
• It returns one of the following errorcodes:
  • 0: The file was recognized and tested without serious errors
  • 1: Unable to open the file
  • 2: JSON errors were issued
  • 3: No valid struct was found to test the file
  • 4: The file was recognized and tested but serious errors were found

After the name you can give a reportlevel which is an integer addition of one or more of the following values. Or alternatly you can substract undesired levels from -1 with -1 meaning report everything. If no value is supplied (1+2+4+8+16+128+256) = 415 or (-1-32-64-512) = -609 is used.

• 1: Missing required keys
• 2: Errors on required values and on either selection
• 4: Errors on not required keys
• 8: Missing sugested keys without a default
• 16: Missing sugested keys
• 32: Missing optional keys without a default
• 64: Missing optional keys
• 128: Unused keys
• 256: Unknown keys
• 512: Report on either selection (without errors)

See the DataTreeGrab 1.2 wiki for more detail.

Testing the sourcefiles and their output

As of release beta-1.0.4-p20161015 there is a script: tv_grab_test_source.py in the package that can perform several tests. See tv_grab_test_source.py --help for the most up to date option list. The options used to set a variable can also be set in ~/.json_struct/tv_grab_test.conf and you can create a default one with: tv_grab_test_source.py --config-file. All reports and a log-file are by default placed in ~/tv_grab_output/ but that directory can both be changed in the config-file or through an option.

Except for the channels modus you do need a configuration file in ~/.xmltv. Before performing a test first the grabber_datafile and the relevant sourcefile is tested with tv_grab_test_json.py placing the output in the report directory.
Use an editor capable of collapsing trees to view the output. This makes it more readable. I use Scite with the language set to Python and when occasionally I work under Windows Notepad++.

The possible options / config values are:

source_dir

Usage: Commandline and config
Commandline: --source-dir
Default: ~/.xmltv/sources

The directory to look for source-files. Be aware that the name of a downloaded source-file is appended with its version before the .json extension. For these tests you need one without the version-number in the name.

grabber_file_dir

Usage: Only configuration
Default: The value set as source_dir

The directory to look for the grabber-data-file. You only need to set it if it differs from the source_dir.

#####grabber_name Usage: Commandline and config
Commandline: --grabber-name
Default: None

The name of the grabber, eg. tv_grab_de

test_modus

Usage: Commandline and config
Commandline: --test-modus
Values: channels| base| detail|lineup
Default: channels

The current four test modi are:

• channels: Test the "channels" or "base-channels" data_def and report on lineup changes.
• base: Test the "base" data_def
• detail: Test the "detail" data_def, but only if the source is marked as a detail-source.
• lineup: Test all sourcefiles with tv_grab_test_json.py and report on any lineup changes. This lineup test is more complete then the one performed by the "channels" modus as it in essence performs a --configure without writing to the configuration. It gives the same output as you get when you set your grabber through InfoFiles to mail you on every run, but then using your local source-files you are working on.

offset

Usage: Commandline and config
Commandline: --offset
Values: 0-14
Default: 0

The day to run the "base" test on

sourceid

Usage: Commandline and config
Commandline: --source-id
Values: any valid integer sourceid
Default: 1

The source to run the "cannels", "base" or "detail" test on

chanid

Usage: Commandline and config
Commandline: --chanid
Default: None

The channel to use for the "base" test. The coresponding channelid for the source is extracted from "source_channels". It does not need to be set active in your configuration. For the test-run this channel is set active with exclussion of all other channels.

detailid

Usage: Commandline and config
Commandline: --detailid
Default: None

The detailid to use for testing a "detail" page. This is the value you on the base-page extract into the detail_url keyword. If no value is given a base-page is extracted and a list of found programmes with a detail_url value set is given to select from.

report_dir

Usage: Commandline and config
Commandline: --report-dir
Default: ~/tv_grab_output

The directory to put all reports

report_level

Usage: Commandline and config
Commandline: --report-level
Values: see Testing the datafiles
Default: 511

The report-level to use on running tv_grab_test_json.py

help

Usage: Commandline
Commandline: --help

Return a description of all possible options

version

Usage: Commandline
Commandline: --version

Return the API version.

config-file

Usage: Commandline
Commandline: --config-file

Create a default config file: ~/.json_struct/tv_grab_test.conf or update the current one to the (through options) set values.

show-sources

Usage: Commandline
Commandline: --show-sources

List all known sources.

show-detail-sources

Usage: Commandline
Commandline: --show-detail-sources

List all known Detail sources.

show-logo-sources

Usage: Commandline
Commandline: --show-logo-sources

List all known Logo sources.

Coming in the near future

• Extension of the functionality of the tv_grab_test_json.py script.
• Extension of the tests that can be performed with .tv_grab_test_source.py
• A graphical GTK program to add and update the language files.

Coming in the further future

An integrated graphical frontend for testing and viewing output. The basic framework is there in the API, but it will take a longer effort to implement.