Dump manager#

The dump manager is in charge to collect data and storages into an archive.

class diskette.core.dumper.Dumper(apps, executable=None, storages_basepath=None, storages=None, storages_excludes=None, logger=None, indent=None)[source]#

Dump manager is in charge of storing application model objects, return serialized datas and storage files to dump them.

Parameters:

apps (list) – List of dictionnaries, each dictionnary is a data dump definition. Each dictionnary will be turned to DrainApplicationConfig or ApplicationConfig object, depending it is a drain or not.

Keyword Arguments:

executable (string) – A path to prefix commands, commonly the path to django-admin (or equivalent). This path will suffixed with a single space to ensure separation with command arguments. This is only used with command.
storages_basepath (Path) – Basepath for reference in some path resolution. Currently used by storage dump to make relative path for storage files. On default this is based on current working directory. If given, the storage paths must be in the same leaf else this will be an error.
storages (list) – A list of storage Path objects.
storages_excludes (list) – A list of patterns to exclude storage files from dump.
indent (integer) – Indentation level in data dumps. If not given, dumps won’t be indented.
logger (object) – Instance of a logger object to use. Logger object must implement common logging message methods (like error, info, etc..). See diskette.utils.loggers for available loggers. If not given, a dummy logger will be used that ignores any messages and won’t output anything.

get_diskette_version()[source]#

Shortand to return diskette version.

Useful with manifest building so it can be easily mocked from tests.

Returns:: The version string.
Return type:: string

get_drain_exclusions(apps, drain_excluded=False)[source]#

Get all model labels that should be excluded from a Drain to respect drainage policy.

Parameters:: apps (list) – List of ApplicationConfig or DrainApplicationConfig objects.
Keyword Arguments:: drain_excluded (boolean) – If True, the excluded models are also returned for application which allows it with their allow_drain option. If False the excluded model won’t be included. Default is False.
Returns:: List of FQM labels.
Return type:: list

load(apps)[source]#

Load application objects from given definitions.

Parameters:: apps (list) – List of dictionnaries where each one is an application.
Returns:: List of application objects built from given definitions.
Return type:: list

dump_options()[source]#

Build a dictionnary of options for each application.

By option we means dump options given to dumpdata.

Returns:

List of dictionnaries, each dictionnary is a payload of application: options.

Return type:

list

payload()[source]#

Build a dictionnary of configuration payload for each application.

Returns:

List of dictionnaries, each dictionnary is a payload of application: definition parameters.

Return type:

list

build_commands(destination=None, indent=None)[source]#

Build dumpdata command line for each application.

Keyword Arguments:

destination (string or Path) – Destination file where to write dump if given. The file will be created by the dump command when executed, not during this method.
indent (integer) – Indentation level for dump data.

Returns:

List of tuples for processed applications, each tuple contains: firstly application name then the built dump command.

Return type:

list

dump_data(destination=None, indent=None, check=False)[source]#

Call dumpdata command to dump each application data.

Keyword Arguments:

destination (string or Path) – Destination file where to write dump if given. The file will be created by the dump command when executed, not during this method.
indent (integer) – Indentation level for dump data.
check (boolean) – Perform operations writhout writing or querying anything.

Returns:

List of tuples for processed applications, each tuple contains: firstly application name then the command output.

Return type:

list

validate_applications()[source]#

Call validators from all enabled application model objects.

Raises:

ApplicationRegistryError – An error with all possible collected errors if
there is any. –

format_archive_filename(filename, with_data=False, with_storages=False)[source]#

Format archive filename depending features.

Keyword Arguments:

filename (string) – Filename to use instead. It must end with .tar.gz. Filename format should be like diskette{features}.tar.gz where features pattern can include either _data, _storages or both depending enabled dump kinds.
with_data (boolean) – Enable dump of application datas.
with_storages (boolean) – Enable dump of media storages.

Returns:

Formatted filename with features.

Return type:

string

build_dump_manifest(destination, data_path, with_data=True, with_storages=True)[source]#

Build dump JSON manifest.

Example of built file (real build is not indented):

{
    "version": "0.0.0",
    "creation": "2024-01-01T12:12:12",
    "datas": [
        "data/djangocontribsites.json",
        "data/djangocontribauth.json"
    ],
    "storages": [
        "var/media"
    ]
}

Note

Involves relative path resolving so it implies that storage paths are proper children of given destination path (that is removed from lead of storage paths).

So storage paths must all start with a starting portion of value from Dumper attribute storages_basepath.

As an example /foo/bar/storage would not be in a ‘storages_basepath’ /ping/ (so it is invalid) but would be (and valid) in /foo or /foo/bar.

Note

Manifest preserve order of registered applications when writing data dump list so it safe for loading them.

Parameters:

destination (Path) – Destination file where to write manifest.

Keyword Arguments:

with_data (boolean) – Enable dump of application datas.
with_storages (boolean) – Enable dump of media storages.

Returns:

Path to the written manifest file.

Return type:

Path

validate()[source]#: Call all validators

make_archive(destination, filename, with_data=True, with_storages=True, with_storages_excludes=True, destination_chmod=420)[source]#

Dump data and storages then archive everything in an archive.

Note

Arguments ‘with_data’ and ‘with_storages’ can not be both disabled, at least one must be enabled else it is assumed as an error.

Parameters:

destination (Path) – Directory where to write archive file.

Keyword Arguments:

filename (string) – Custom archive filename to use instead of the default one. Your custom filename must end with .tar.gz. Default filename is diskette[_data][_storages].tar.gz (parts depend from options).
with_data (boolean) – Enable dump of application datas.
with_storages (boolean) – Enable dump of media storages.
with_storages_excludes (boolean) – Enable usage of excluding patterns when collecting storages files.
destination_chmod (integer) – File permission to apply on archive files and also on destination directory if it did not exists. Value must be in an octal notation, default is 0o644.

Returns:

Path to the written archive file.

Return type:

Path

make_script(destination, with_data=True, with_storages=True, with_storages_excludes=True)[source]#

Create shellscript command lines to dump data.

Parameters:

destination (Path) – Directory where to write archive file.

Keyword Arguments:

with_data (boolean) – Enable dump of application datas.
with_storages (boolean) – Enable dump of media storages.
with_storages_excludes (boolean) – Enable usage of excluding patterns when collecting storages files.

Returns:

All commands to dump data, each command on its line with a previous: comment line with the dump name.

Return type:

string

check(destination, filename, with_data=True, with_storages=True, with_storages_excludes=True, destination_chmod=420)[source]#

Check what would be done.

Parameters:

destination (Path) – Directory where to write archive file.

Keyword Arguments:

filename (string) – Custom archive filename to use instead of the default one. Your custom filename must end with .tar.gz. Default filename is diskette[_data][_storages].tar.gz (parts depend from options).
with_data (boolean) – Enable dump of application datas.
with_storages (boolean) – Enable dump of media storages.
with_storages_excludes (boolean) – Enable usage of excluding patterns when collecting storages files.
destination_chmod (integer) – File permission to apply on archive files and also on destination directory if it did not exists. Value must be in an octal notation, default is 0o644.

Returns:

Path to the written archive file.

Return type:

Path