backupagentnq - cloud backup enqueuer agent
backupagentnq - cloud backup enqueuer agent#
backupagentnq is the cloud backup enqueuer agent. It is responsible for
identifying the files to backup and enqueues their pack instruction files into
Spool/Work directory. Its companion
responsible for dequeuing and submitting the pack instruction files to the
Pack instruction files#
Pack instruction files are the fundamental unit of backups. Each backup file will be converted to a series of instructions, which the cncop cloud network will follow to create a cloud version of the file.
Pack instruction files come in two variations; pack-add and pack-update. A pack-add instruction file provides a complete copy of the source file. A pack-update instruction file provides directions for modifying a previous backup file to match the changes stored on a prime.
backupagentnq agent operation has three distinct phases
Peek - The peek phase is where the agent identifies those files that have either changed, been added, or been deleted from the local device and need backup to the cloud. Each planned action is recorded in the work queues database, e.g.,
Pack - The pack phase is where the agent generates pack instruction files using the
cncop.packgen2system and enqueues them into the
Prune - The prune phrase is where the agent deletes files from the cloud that have been deleted from the local device.
From time to time, a user will reorganize their files and folders. An
example is at the end of a reporting quarter or the end of a year when they
might rename an old document tree to include the last year, e.g.,
20NN-Proposal/. The agent needs to identify this
type of folder maintenance; otherwise, it would interpret the action as a mass
of DELETES and ADDS. And the resulting batch of ADDS could cause a lengthy
backlog that could take days or weeks to re-upload to the cloud.
To avoid this, the peek phase searches for move promotions. This involves scanning the ADD and DELETE work queues for potential matches and removing and replacing these two operations with a MOVE and an UPDATE operation if a match is detected.
The full details of move promotion and what attributes it considers are described and protected by US Patent # 7,668,880
Skip backup when nothing has changed#
To conserve Internet bandwidth, the
backupagentnq will skip
performing a cloud peek when it determines nothing has changed on the local
device. Instead, it employs a last_write check system that allows it to know for
each job when the newest file was last written, and if nothing more recent has been
added or changed, the
backupagentnq will end without peeking the cloud.
--forcepeek command line option to override this check and force a
complete peek of the local device and the cloud.
Protections against unintended deletes#
backupagentnq will limit the number of deletes it will perform in
any single backup run to a delete collar specified in the job (default 1,500).
This is to protect against accidental mass deletes. Use the
command line options to suppress the delete collar and perform all scheduled
Two singleton protections are embedded in the backupagentnq system; a joblock and an applock.
The joblock protection prevents multiple running copies of the same backup job. Any attempt to run a second copy of the job will cause the agent to wait for the first instance to finish and release its joblock.
The applock will protect against running more than one instance of the
backupagentnq software. The reason is to prevent excessive write-lock
contention for the work database and to limit the number of writers to the
default logging file (ex:
Logs/NQ-YYYY-MM-DD.log) to a single writer.
When there is a single long-running
backupagentnq agent, the use of the
--noapplock option will disable the applock protection and allow another
copy of the software to run. When invoked with this option, the log file prefix
will change to
NQU (the extra
U is applied to indicate it was invoked
--noapplock does not eliminate the joblock singleton pattern, which
is to say, you cannot use this option to run the same job more than once.
Command line options#
backupagentnq [-?] [-d] [--debug2] [--debug3] [-c CONFIG] [-n]
[--nomaxdelete] [--noapplock] [--skipmoves] [--skipdeletes] [--skippeeksource]
[--skippeekcloud] [--usefork] [--nofork] [--usevss] [--forcepeek] [--nicld]
[--suppressmissing] [--chunksize CHUNKSIZE] [--savepeek SAVEPEEK] [--peeklog]
[--nolinks] [job [job ...]]
- ?, -h, --help#
Display help and exit
- -d, --debug#
Generate diagnostic logging. The output is directed to the
Spool/Logsfolder and can be viewed using the cncop watcher script.
Advanced diagnostic logging. Will emit verbose details of pack file generation. This option is intended for developers and not recommended for production.
Extreme diagnostic logging. Not recommended for production use.
- -c CONFIG, --config CONFIG#
Alternate configuration file.
- -n, --dryrun#
Run agent in reporting mode. Does not perform a backup, but instead reports on what it would do. Note: use of this option will invalidate our source peek information, and cause the next invocation of this agent to perform a full cloud peek.
Removes the maximum delete collar and allows all cloud deletes to take place.
Disable application locks. The default is to prevent more than one copy of
backupagentnqfrom running at a time. If a second copy tries to run, it will wait until the first one finishes before running. When the agent is invoked with
--noapplock, the application lock is disabled and another
backupagentnqcan run. If invoked with
--noapplock, the agent will write it’s log information to
Instruct the peek phase to skip checking for MOVED files. This option is only useful when there are a large number of ADD’s and DELETE’s, and scanning these two work queues for MOVE matches could consume excessive time and memory.
Do not delete files from the cloud. This is useful for the 1st sync with the cloud after it’s been seeded from a shuttled hard drive.
Do not peek the source filesystem. This option is primarily for diagnostics and should not be used in production.
Do not peek the cloud. This option will reuse the most recently downloaded cloud peek. It should only be used for diagnostics.
Override fork detection logic and force peek to use filesystem forks. This option is for development use and not recommended for production.
Override fork detection logic and disable the use of filesystem forks. This option is for development use and not recommended for production.
Backup using VSS. This option will cause the agent to make a new VSS snapshot and backup the specified path using the snapshot.
Override the job’s newness detection system and force the agent to perform a cloud peek.
Suppress logging of
PEEK-NICLDmessages. If include filters are used, this agent will log all files that do not match the filter, which, if the source path is large, can cause excessive log messages. This message disables this logging.
Suppress warnings of missing sourcefiles. If the filesystem is undergoing rapid changes while the backup job is running, some files may disappear after they’ve been scheduled for backup. This option suppresses generating job warning messages about any missing sourcefiles.
- --chunksize CHUNKSIZE#
cncopagentprocess calls the cloud enqueuer and dequeuer agents with the same command line parameters. The enqueuer agent ignores this parameter.
- --savepeek SAVEPEEK#
Save the peek contents received from the cloud a local xml file. This option is for debugging cloud responses and likely only useful for s/w developers.
--savepeek, this will cause the peek data to be preserved, but rather than specifying the file’s name, the system will select a unique basename and store this file in the cncop log directory with the prefix NQPK. This feature enables debugging infrequent peek errors that reoccur with unpredictable frequency.
Do no backup file system links. Use of this option will cause the agent to skip backing up symlinks and soft links. Without this option, the agent will “follow” the link and backup what the link points to.