Task queue processor

What is the dataskq processor?

DirectAdmin processes many different tasks, some of them are immediate while others are scheduled to be processed by an internal processor - the task queue manager (/usr/local/directadmin/dataskq). It is executed by main directadmin service every minute (can be changed with dataskq_run_interval config option).

Similar tasks are grouped and executed only once. For example, adding new domain, subdomain, enabling SSL requires a reload of the WWW server. If there are multiple tasks queued to reload WWW server, reload would be executed only once. This means that you have to wait at least 1 minute after doing some vague changes to the server before the dataskq processor is run.

The task queue is stored in the /usr/local/directadmin/data/task.queue file. When queue processing is finished, the file is removed. It is normal to find this file absent on the system.

Tasks can be input into the task.queue file manually to be processed by dataskq. It is just the syntax that should be known for this. Most common examples:

restart directadmin: echo "action=directadmin&value=restart" >> /usr/local/directadmin/data/task.queue
rewrite user's WWW server configuration: echo "action=rewrite&value=httpd" >> /usr/local/directadmin/data/task.queue
update DirectAdmin service: echo "action=update&value=program" >> /usr/local/directadmin/data/task.queue
process nightly tally: echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue

Dataskq could be launched manually without waiting for the cronjob to process it. It also accepts different debug levels:

/usr/local/directadmin/dataskq d80

Where d80 is debug level. Commonly used levels are d80, d400, d800, d2000.

What is the task.queue.cb file

There could be another task.queue file /usr/local/directadmin/data/task.queue.cb which is always parsed and mostly is filled by the custombuild script. Could be used same way like the 'real' task.queue.

The "feature" part is that the dataskq can now be called like this:

/usr/local/directadmin/dataskq d2000 --custombuild

With the --custombuild option passed to the dataskq (with or without debug mode), this tells the dataskq to skip all other tasks, and only process the task.queue.cb file.

This means:

The task.queue.cb file is executed.
No other task.queue or task.queue.da file is executed
Services are not checked if they're running
Partition usage is not checked
Brute force scanner is not run
The DA login history counter is not checked (for port 2222)

Only one instance of the dataskq at a time

Since dataskq is being run every minute if things like backups are issued at different times, say 5 minutes after the first run of a backup, then you'll end up having 2 backups being created at the same time, which can generate slow performance because of the drive being forced to bounce between 2 locations on disk simultaneously.

The solution for that is to only have 1 instance of the dataskq ever running at one time. Note that the consequence to that is that any actions that needs to be run while a backup is executing will not be processed. E.g., service monitoring, service restarts, or other task.queue commands. They will be stored in the task.queue file and executed once the first instance of the dataskq has finished.

It is possible to limit the number of concurrent dataskq process with the dataskq_max_instances config parameter. Setting it to 1 will ensure no new dataskq processes will be started until the already running one will finish its tasks.

da config-set dataskq_max_instances 1 --restart

Note: We do not recoomend changing the default. Despite being slower concurrent dataskq processes ensures all changes and background tasks are executed in a timely manner. Please use this config parameter only as a last resort. To minimize the potential impact it could be set to 3 or 5 instead of 1.

I want the dataskq to run more than once per minute

It can be controlled with the dataskq_run_interval config option. Running dataskq every 30 seconds:

da config-set dataskq_run_interval 30s --restart

The dataskq is causing 100% CPU

If you're noticing your system load is on the rise, and you can see that the "dataskq" binary is running at the top of the list (in "top"), this guide will help you debug what it's doing.

Check /var/log/directadmin/system.log to see if the tally is being run. If Users continue to be added to the log (the logs are doing something), then it's likely just the nightly tally, which is normal (assuming the log continues to grow and isn't just stuck on one User).
The first thing to do, is to simply ask the dataskq what it's up to. To do this, type:

killall -USR1 dataskq
tail -n 10 /var/log/directadmin/errortaskq.log

What this will do is dump the running dataskq's current process location to the errortaskq.log. What that log output says will determine what to do next.

If the output makes any reference to Maildir along with a path, then what it likely means is that the mentioned path contains an over-sized inbox. Check that inbox and delete the messages, if the email user doesn't seem to be deleting them.
If the output makes any reference to brute_force or some related file, then the cause is likely the dataskq chewing on the system logs with many entries.

First, ensure you're using the latest version of DirectAdmin.
Then check:

cd /usr/local/directadmin/data/admin
ls -la brute_force*

We're checking for a large brute_log_entries.list file. If it seems to be "large" (say, over a Meg in size), then that can slow things down. You can safely delete "brute_log_entries.list", as it's not part of the Brute Force Monitor's (BFM's) counting...it's only for your own purposes, showing you what each attack was. DA will recreate this file automatically.

If the brute_log_entries.list continues to grow and you'd like it to keep itself smaller, go to Admin Level -> Admin Settings and lower the value for Clear failed login attempts from log X days after entry was made. to something around 2 days.
Also, increasing the values for Notify Admins after an IP has X login failures on any account and Notify Admins after a User has X login failures from any IP. will reduce the number of entries made into the brute_log_entries.list file.

My monthly bandwidth is not resetting

The monthly bandwidth reset is done at 4:20am on the first day of the month. If you find that this did not happen (allow about an hour for the dataskq to run, pending the number of sites), then there are a few steps you can take to find out why, and reset it manually.

The first thing to check is the cron logs to ensure that the reset was actually issued. Check the /var/log/cron for the 4:20am timestamp for the 1st day of the month. You should see something like:

echo 'action=reset&value=all' >> /usr/local/directadmin/data/task.queue

If you don't see that command, then you'll want to check this guide to ensure that crond is running and the /etc/cron.d/directadmin_cron file is working correctly.

If you do see the above command in your cron log, then the next step is to check /var/log/directadmin/system.log for the entry that looks something like this:

2019:04:01-04:20:01: Reset all command Received

If you don't see that log entry, then you'd again want to use the same guide as above to debug the issue.

If you do see the above log entry, then scroll just below it to confirm that each user is mentioned. If not, then likely the users are not all set in the users.list file. You can use this guide to reset all users.list files.

At this stage, hopefully you would have figured out why the reset didn't happen, and have fixed it so it does reset all users' bandwidth next time. However, the reset still wouldn't have happened for the past reset. So to do that, you'd use the cleanreset task.queue command. Type:

echo "action=cleanreset&value=all" >> /usr/local/directadmin/data/task.queue
/usr/local/directadmin/dataskq d
echo "action=tally&value=all" >> /usr/local/directadmin/data/task.queue

which should reset things from the start of the month without losing data. Manually doing the regular reset mid-month will lose current data that you still want, hence we use the cleanreset, which scans all datestamps to ensure it doesn't delete anything it shouldn't.

If you didn't figure out why the reset didn't happen, you may also want to check for other competing tasks. For example, if the server is rebooted at the time of the reset, that could prevent it from working. Also, the dataskq is also used by other tasks such as backups, so check to see if you've got any large backups running near the same time as the reset, and check /var/log/directadmin/errortaskq.log for any errors relating to them.

Make the bandwidth update more often

Cron is the tool used to control the action of the stats tally. It is performed once 10 minutes after midnight every day.

If you want to run it more often a new cron script can be created with additional requests to perform tally. Example of /etc/cron.d/directadmin_cron_custom contents:

10 6 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
10 12 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
10 18 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue

Would execute tally at 00:10, 06:10, 12:10 and 18:10. Note that 00:10 is default tally execution time, additional times are specified in new file.

One important thing to remember is that for very large servers, with upwards of 1000 domains, the tally can take a significantly longer time to run (up to 2 hours), so if you run a large server, you may not want to run the tally too often.

Once you've set the value you want, save the file and reload the cron daemon:

service crond restart

WWW server graceful restarts after the nightly tally

After the nightly tally runs, the dataskq will issue a graceful restart to the WWW server to both re-open logs and reload configs in case accounts have been suspended/unsuspend:
http://www.directadmin.com/features.php?id=980open in new window
If you wish to disable this (not recommended), you may use this:
http://www.directadmin.com/features.php?id=1020open in new window
Use the tally_post.sh to take some special action that you want (note that it's called just before the graceful restart, as the restart is sent to the task.queue to be processed in the next minute/dataskq run).

How to delete domains from the command line

Domain deletion could be passed to the dataskq process with this command:

echo "action=delete&value=domain&requestedby=admin&select0=domain.com(&select1=domain2.com...)" >> /usr/local/directadmin/data/task.queue

Where options are:

requestedby=admin : controls where the result Message is sent to within the Message System, where the result is either success or fail. It does not have to be the owner of the domain, but can be. The actual domain owner is looked up for each domain.
select0=domain.com : the domain to be deleted.
select1=domain2.com : optional other domains, for any User, does not need to be under the same User. (select2=, select3=, etc.)

How to link IPs trough the task.queue

Related to interface Admin Level -> IP Manager -> click on an IP -> Link IP, this can now be done through the task.queue.

Sample:

echo "action=linked_ips&ip_action=add&ip=1.2.3.4&ip_to_link=1.2.3.5&apache=yes&dns=yes&apply=yes" >> /usr/local/directadmin/data/task.queue

The command is the same as the form in CMD_IP_MANAGER_DETAILS?ip=1.2.3.4, except we move the form action=add to ip_action=add, as the dataskq action needs to be broader.

How to Message Users From the Command Line

To message a specific account, list of accounts or all accounts (Admins+Resellers+Users), the task.queue could be used. Format is as follows:

action=notify
value=users
users=all|select1=fred&select2=george&select3=gary
subject=Your Subject
message=Content of your Message

where the "users" variable can be "all" (meaning DA will message all accounts on the server) or can be a list of accounts that are url-encoded with select1, select2.

For a single account, just use:

users=select1=gary

Note that the = and & characters in the users list must be url encoded with hex (%3D and %26, respectively). Also, if you want to include a newline in the message, use %0A instead.

A sample echo command is:

echo 'action=notify&value=users&subject=Hello World&message=Hello,%0A%0A  This is a message.&users=select1%3Dadmin%26select2%3Dgary' >> /usr/local/directadmin/data/task.queue

How to send notification to the Admin Message System

This is a very simple new task.queue command to send Admin notices.

action=notify
value=admin
subject=Your Subject
message=Content of your Message

It should be URL encoded (mainly if you want newline characters: %0A) so that it can be posted to the task.queue.

Here's an example:

echo "action=notify&value=admin&subject=hello this is my subject&message=this is the notification%0Awith a new line." >> data/task.queue.cb; ./dataskq d10 --custombuild

Note: the above sends to the task.queue.cb file, and calls the dataskq with the --custombuild option.

This skips all other typical dataskq commands, so you don't accidentally process a backup or the tally.

How to run bandwidth only tally

You may force dataskq to run bandwidth only tally using the following command:

echo "action=bandwidthtally&value=all" >> /usr/local/directadmin/data/task.queue

Alternatively to process per-user:

echo "action=bandwidthtally&value=all&value=user&type=username" >> /usr/local/directadmin/data/task.queue

How to run a quota only tally

When you delete a domain, the disk usage and totals are recounted for the domain as the deletion actually runs a 'mini' tally for quota only, like:

echo "action=quotatally&value=bob&type=user" >> /usr/local/directadmin/data/task.queue

You may recalculate the 'quotatally' for all users like so:

echo "action=quotatally&value=all" >> /usr/local/directadmin/data/task.queue

In the system.log, the usual tally entries will show up, but with this, it will show the mask, e.g.,

Tally User admin Begin: mask 20

This lets you know what the tally is actually looking at. If no mask is present, e.g.,

Tally User admin Begin

then everything is checked.

The mask layout is as follows:

#define TLY_MASK_WEBALIZER 1
#define TLY_MASK_BANDWIDTH 2
#define TLY_MASK_DISKUSAGE 4
#define TLY_MASK_LOGROTATE 8
#define TLY_MASK_TOTALS 16
#define TLY_MASK_SUSPEND 32

so, that means a mask of 20 = 16+4, so we have totals + quotas in the above example.

How to set quota limits for users

This command will go through each user, and set the quota limits in the system quotas. Users with unlimited quotas will also have their quota limit set to 0 (unlimited):

echo "action=rewrite&value=quota" >> /usr/local/directadmin/data/task.queue

Or per user:

echo "action=rewrite&value=quota&user=bob" >> /usr/local/directadmin/data/task.queue

How to rewrite a DNS zone

The domain key could be used to rewrite a DNS zone for only that domain:

echo "action=rewrite&value=named&domain=domain.com" >> /usr/local/directadmin/data/task.queue

Also, an optional key could be set to not update the DNS zone's serial number:

update_serial=no

If update_serial is not passed, yes is assumed.

echo "action=rewrite&value=named&domain=domain.com&update_serial=no" >> /usr/local/directadmin/data/task.queue

How to recreate spambox settings for mailboxes

When you make change in your SpamAssassin settings, each time you click "Save", DA will recreate/set the ownership/permissions on each spambox file.

If you have a large number of pop accounts, this can take long enough that the timeout is reached. Moving this task to the task.queue file to be run in the background will allow things to run quicker in the foreground in DA.

The actual task.queue command:

echo "action=defaultspam&domain=domain.com&username=user" >> /usr/local/directadmin/data/task.queue

A command to rewrite spam filters is:

echo "action=rewrite&value=filter" >> /usr/local/directadmin/data/task.queue

How to reset monthly stats

In the case that the monthly reset did not run, an admin can run:

echo "action=cleanreset&value=all" >> /usr/local/directadmin/data/task.queue

in order to only remove the previous month's data.

This will do a standard reset of bandwidth.tally files and user.usage files, but will not follow with a tally.

You'd need to run the full tally command afterwards to recount the correct stats for Reseller Level and Admin Level counting:

echo "action=tally&value=all" >> /usr/local/directadmin/data/task.queue

How to rewrite virtual user files

To rewrite/convert the /etc/virtual/domain.com/passwd files, you may use following command:

echo "action=rewrite&value=email_passwd&user=fred" >> /usr/local/directadmin/data/task.queue

Where fred is desired user that owns domain.com.

Or, to rewrite all, run:

echo "action=rewrite&value=email_passwd" >> /usr/local/directadmin/data/task.queue

How to rewrite apache/nginx configs for single user

Instead of rewriting configs for all users, save time by rewriting a single account's config as follows:

echo "action=rewrite&value=httpd&user=fred" >> /usr/local/directadmin/data/task.queue

List of other taskq commands

A list of other useful taskq commands that one could echo into the task.queue file are outlined below.

Regenerate vacation config files, first for all, and then for a single domain:

action=vacation&value=all
action=vacation&value=steve.com&type=domain

Regenerate email filters, first for all, and then for a single user:

action=rewrite&value=filter
action=rewrite&value=filter&user=bob

Regenerate email passwd files, first for all, and then for a single user:

action=rewrite&value=email_passwd
action=rewrite&value=email_passwd&user=fred

Regenerate email aliases, first for all, and then for a single user:

action=rewrite&value=email_aliases
action=rewrite&value=email_aliases&user=fred

Regenerate /etc/virtual/domainips and helo_data, first for all, and then for a single domain/ip:

action=rewrite&value=domainips
action=rewrite&value=domainips&domain=domain.com
action=rewrite&value=helo_data
action=rewrite&value=helo_data&ip=1.2.3.4

Run sync to cluster with 10 retries (in case one server is unreachable):

action=sync&type=cluster&value=domain.com&count=10

Run sync to cluster email data for the user fred to host 1.2.3.4 (failed host), with 10 retries and the longrequest URL encoded POST from the original client request:

action=sync&type=cluster&value=email&username=fred&host=1.2.3.4&count=10&request=long%request%to%set%email%account

Run domain verification process of /etc/virtual/domainowners to /usr/local/directadmin/data/users/fred/domains/:

action=confirm&type=domains.list&user=fred

Check if user/file exists and ensure the lines in Apache configs are correct based on that:

action=confirm&type=httpd.conf&value=/usr/local/directadmin/data/users/fred/httpd.conf

Regenerate the users' pureftpd database:

action=rewrite&value=pureftp_db

Regenerate dkim 1) for all domains, 2) for a single domain, and 3) for a single domain and add the key to the domain's DNS:

action=rewrite&value=dkim
action=rewrite&value=dkim&domain=domain.com
action=rewrite&value=dkim&domain=domain.com&dns=yes

Forcefully run BFM to scan Apache logs 1) for all domains, 2) for a list of users, and 3) for a list of domains:

action=rewrite&value=brute_force_scan_apache_logs
action=rewrite&value=brute_force_scan_apache_logs&users=...userslist...
action=rewrite&value=brute_force_scan_apache_logs&domains=...domainslist...

Update/check the panel license:

action=update&value=license
action=check&value=license

Update the DirectAdmin panel itself:

action=update&value=program

Creating backups:

action=backup&type=user&value=john
action=backup&owner=reseller&type=reseller&value=multiple&when=now&where=local&who=all
action=backup&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=george&select2=eddy.. [ftp_ip=1.2.3.4&ftp_username=asdf&ftp_password=base64enc&ftp_path=/]
action=backup&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=buger
action=backup&value=now&type=system
action=backup&type=sitebackup&value=bob&password=pass&select0=ftp&select1=email, etc (backup options)

Restoring from a backup:

action=restore&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=george.tar.gz&select2=eddy.tar.gz.. [ftp_ip=1.2.3.4&ftp_username=asdf&ftp_password=base64enc&ftp_path=/]

Notify user 'bob' about blacklisted IP '1.2.3.4':

action=notify&value=ip_blacklist&ip=1.2.3.4&time=12345678&username=bob

Regenerate the caches used throughout the DirectAdmin panel:

action=cache&type=popquota
action=cache&type=popquota&value=domain.com
action=cache&value=showallusers
action=cache&value=showallusers&user=bob
action=cache&value=showallusers&select0=bob&select1=fred&...
action=cache&value=safemode
action=cache&value=safemode&domain.com=username&domain2.com=user2

Add or delete DNS records:

action=dns&do=add&domain=es5.com&type=A&name=taskqrec&value=1.2.3.4&ttl=500
action=dns&do=delete&domain=domain.com&type=A&name=some.thing.&value=1.2.3.4
action=dns&do=delete&domain=domain.com&type=A&name=some.thing.&value=* //deletes any value.
action=dns&do=delete_zone&domain=domain.com

Delete old mbox files (nowadays Maildir format is in use):

action=delete&value=mbox

Convert mailboxes to use new mail partitioning:

action=convert&value=mail_partition
action=convert&value=mail_partition&smart=yes
action=convert&value=mail_partition&user=fred
action=convert&value=mail_partition&user=fred&smart=yes
action=delete&value=old_mail_partition
action=delete&value=old_mail_partition&user=fred

Delete users, delete domains, and lastly, delete databases:

action=delete&value=account&requestedby=admin&select0=bob(&select1=fred...)
action=delete&value=domain&requestedby=admin&select0=domain.com(&select1=domain2.com...)
action=delete&value=test_dbs

Switch from one IP address to another:

action=ipswap&old=1.2.3.4&new=4.3.2.1

Update plugin (CustomBuild GUI plugin example):

action=plugin&value=update&username=admin&select0=custombuild

# Task queue processor

# What is the dataskq processor?

# What is the task.queue.cb file

# Only one instance of the dataskq at a time

# I want the dataskq to run more than once per minute

# The dataskq is causing 100% CPU

# My monthly bandwidth is not resetting

# Make the bandwidth update more often

# WWW server graceful restarts after the nightly tally

# How to delete domains from the command line

# How to link IPs trough the task.queue

# How to Message Users From the Command Line

# How to send notification to the Admin Message System

# How to run bandwidth only tally

# How to run a quota only tally

# How to set quota limits for users

# How to rewrite a DNS zone

# How to recreate spambox settings for mailboxes

# How to reset monthly stats

# How to rewrite virtual user files

# How to rewrite apache/nginx configs for single user

# List of other taskq commands