Configuring a New Source Collector

Modified on Thu, 30 Jul at 9:15 AM

Overview

This guide explains how to configure and run a Source Collector in the Varibill User Interface.

It covers user interface configuration. For sending data directly to the Source Collector API, see Source Collector API Configuration.

Note: Before configuring a collector, it helps to understand how Varibill classifies collected data. See Source Collector Classification and Definitions.

Rated and Unrated Data

A Source Collector submits one of two kinds of data, and this choice affects how you configure it:

Unrated data: raw usage or quantity records. Varibill applies your selling rates to calculate the billed amount.
Rated data: pre-calculated records where you supply the cost, price, and quantity yourself.

Choose the type that matches your data source. Unrated collection relies on selling rates being configured in Varibill for the products involved.

Accessing the Varibill User Interface

Source Collectors are configured through the Varibill User Interface.

After a successful login, you are presented with the Varibill Dashboard:

Note: If this is your first time logging in, see the First Login and Access guide for help signing in.

Navigate to Source Collectors

To add a new data source using the Source Collector:

Hover over the Navigation menu.
Go to Billing Data.
Select Source Collector.

Navigation Flow to Source Collector Editor

If Source Collection has never been run on your account, the following default file types are visible:

Contract Service
File Upload
Manual Entry
System Filled Data

Note: These are default collection files and cannot be edited.

Source Collector Process Flow

At a high level, the process follows this flow:

Navigate to Billing Data → Source Collector.
Click + New Source Collector to open the Source Collector Editor.
Configure the collector across its sections: Details, API Credentials, Runtime, Schedule, IP Addresses, Notifications, and Batches.
Click Save to create the system job and commit the configuration.
Run the collector, manually or on its schedule.
Monitor the Dashboard, Job Executions, Source Data Files, and Validation Errors.

Configuring a New Source Collector

Click the + New button.

The Source Collector Editor window opens, where you can configure your data source.

Source Collector Field Data

Details Window

Source (Collector) Name: User-defined, required.
Status:
- Active – for current use.
- Inactive – for future use.
Source Type: System-generated (defaults to Source Collector).
Source Description: Required (maximum 250 characters).
Atomic (checkbox):
- Enabled: all records must succeed or none are committed.
- Disabled: best-effort processing (partial success allowed).
Validation & Billing (checkbox):
- Enabled: included in validation and invoicing.
- Disabled: excluded from billing.
Ignore Record Identifier (checkbox): When enabled, changes to identifiers or descriptions do not create new billing lines.

Tip: Use a clear, recognizable collector name so it is easy to identify on the Dashboard and in Job Executions.

Warning! Identifier configuration affects billing. Incorrect setup may result in:

Duplicate billing
Split billing periods
Unexpected pro-rating

Always validate identifier configuration before enabling production billing, and avoid changes mid-cycle.

See Source Collector Classification and Definitions for how data types and collection behavior impact billing.

Source Collector API Credentials

Tenant Key: System-generated.
Source Key: System-generated (can be regenerated).
Source Secret: System-generated.
Username: System-generated.

Note: These values are used to authenticate when submitting data via the Source Collector API or a self-hosted collector.

Tenant Key and Source Key form the API username.
Source Secret is the API password.

The same values map directly to the parameters used by a self-hosted collector: Tenant Key to -TenantKey, Source Key to -SourceCollectorKey, and Source Secret to -SourceCollectorSecret.

Tip: A Source Collector must be configured before any data can be submitted. These credentials are generated automatically when the collector is created.

Runtime Window

Hosting Provider

Self Hosted: You provide and manage the infrastructure that runs the collector. The collector authenticates with the credentials above and submits data to the Source Collector API.
Varibill Cloud: Select from the available cloud-based collectors managed by Varibill. See the Varibill Cloud Collector Setup Guides for the supported collectors.

Azure Window (Varibill Cloud only)

Azure App: System-generated.
Replica Retry Limit: Number of retry attempts before failure.
Replica Timeout: Time (in seconds) before timeout and retry.
Container App Job Name: Auto-assigned (read-only).
CPU Cores: Processing power allocation. Higher values can improve performance and may increase cost.

Note: Higher CPU core counts can reduce failures but increase cost. Contact Varibill Support for recommendations.

Schedule (CRON Expression) Window

CRON Expression Scheduling Window Example

Execution Mode

Manual: Run on demand.
Scheduled: Runs automatically based on a CRON expression.

CRON field reference:

Minute: 0–59
Hour: 0–23
Day of month: 1–31
Month: 1–12
Day of week: MON–SUN

Example:

0 9 * * MON runs at 9 AM every Monday.

Note: Use an asterisk (*) to represent “every” (for example, * * * * * runs every minute).

Tip: Use crontab.guru to build and test expressions.

IP Address/es Window

Used to whitelist IP addresses for legacy collection patterns. This may be phased out as modern authentication replaces legacy methods.

Notification Recipients Window

Configure the email recipients for failed collection runs.

Note:

Add at least one technical contact to failure notifications.
Missed collections cannot be backdated, and data may be permanently lost.

Batches Window

Batches segment a collection run into smaller sets of records, which simplifies troubleshooting and improves reliability for large data sources. Self-hosted collectors split data automatically into batches of up to 10,000 records per file by default.

Source Data Files and Job Executions

The editor also provides views to monitor and validate a collector after it runs:

Job Executions: Each run of the collector, with status and timing.
Source Data Files: The data files produced or uploaded for the collector, which you can review or remove.
Validation Errors: Issues found when the collected data is validated against your billing setup.

Saving and Running a Source Collector

Click Save to:

Commit the collector configuration.
Enable collector execution.

After a run, Job Execution Files become available for validation and troubleshooting.

Troubleshooting Failed Source Collector Runs

Use the Varibill Dashboard to identify anomalies. Sudden spikes or drops in collected data may indicate a problem.

Use View Details to inspect:

Events
Metrics
Specific tenant and Source Collector performance