Self-Hosted Source Collectors

Overview

A self-hosted Source Collector runs on infrastructure you manage and submits billing data to the Varibill Source Collector API. You choose this option when your data lives in a system Varibill does not collect from directly, or when policy requires collection to run inside your own environment.

Self-hosted collection is built on the Varibill Source Collector Core PowerShell module. The module handles authentication, batching, retries, and submission, so a collector script only needs to gather data and hand it to the module.

Rated and Unrated Data

The module supports two data types:

• Unrated Source Data: raw usage or quantity records. Varibill applies your selling rates to calculate the billed amount.
• Rated Source Data: pre-calculated records where you supply the cost, price, and quantity.

Use unrated data when Varibill should do the rating, and rated data when your source system already calculates the charge.

Prerequisites

• PowerShell 5.1 or later (PowerShell 7 or later recommended).
• A Source Collector configured in the Varibill User Interface, set to Self Hosted.
• The collector credentials from the Source Collector API Credentials section of the editor:
  • Tenant Key
  • Source Collector Key
  • Source Collector Secret
• The Source Collector API URL for your environment.
• Network access from the host to the Source Collector API.

How a Self-Hosted Collector Works

The Core module provides two classes that follow the same workflow:

1. Create a data object with your collector credentials.
2. Add records to it.
3. Save the data to disk.
4. Post the data to the Source Collector API.

The module also provides:

• Automatic batching: records are split into files of up to 10,000 records each by default.
• Staging files: pending uploads use a .rdy extension; completed uploads use .json.
• Response logging: API responses are saved for audit and troubleshooting.
• Configurable storage: data is written to a temporary directory by default, or to a path you specify.

Installing the Core Module

Install from the PowerShell Gallery:

Install-Module -Name Varibill.SourceCollector.Core -Scope CurrentUser

Then import it before use:

Import-Module Varibill.SourceCollector.Core

Configuring Credentials

Assign your credentials and environment URL to variables. The keys come from the Source Collector API Credentials section of the collector editor.

# Confirm the Source Collector API URL for your environment.
$SourceCollectorAPI    = "https://api.varibill.com/"
$TenantKey             = "your-tenant-guid"
$SourceCollectorKey    = "your-collector-guid"
$SourceCollectorSecret = "your-secret"

Submitting Unrated Source Data

Use unrated data to send raw usage that Varibill will rate.

$unrated = New-UnratedSourceData `
    -SourceCollectorAPI $SourceCollectorAPI `
    -TenantKey $TenantKey `
    -SourceCollectorKey $SourceCollectorKey `
    -SourceCollectorSecret $SourceCollectorSecret

# AddUnratedSourceData(Client, Product, Record, UID, Timestamp, Quantity, Tag)
$unrated.AddUnratedSourceData(
    "ClientName",
    "ProductCode",
    "RecordDescription",
    "UniqueID",
    (Get-Date),
    100,
    "OptionalTag"
)

$unrated.SaveData()
$unrated.PostSourceData()

Submitting Rated Source Data

Use rated data when your source system already calculates cost and price.

$rated = New-RatedSourceData `
    -SourceCollectorAPI $SourceCollectorAPI `
    -TenantKey $TenantKey `
    -SourceCollectorKey $SourceCollectorKey `
    -SourceCollectorSecret $SourceCollectorSecret

# AddRatedSourceData(Client, Product, Record, UID, Timestamp, Quantity, Cost, Price, Tag)
$rated.AddRatedSourceData(
    "ClientName",
    "ProductCode",
    "RecordDescription",
    "UniqueID",
    (Get-Date),
    100,
    50.00,
    75.00,
    "OptionalTag"
)

$rated.SaveData()
$rated.PostSourceData()

Using the Example Collectors

The framework ships ready-made collectors you can use as a starting point or run directly. Each collector documents its own parameters, but all of them take the standard credential parameters above.

The VMware vCenter collector, for example, gathers virtual machine resource usage (CPU, memory, and storage) and submits it as unrated data:

./VCenter_0.5.ps1 `
    -SourceCollectorAPI $SourceCollectorAPI `
    -TenantKey $TenantKey `
    -SourceCollectorKey $SourceCollectorKey `
    -SourceCollectorSecret $SourceCollectorSecret `
    -VCenterHost "vcenter.example.com" `
    -VCenterUsername "admin" `
    -VCenterPassword "password" `
    -CPUProductCode "VM-CPU" `
    -MemoryProductCode "VM-MEM" `
    -HDDProductCode "VM-HDD"

Running in a Container

The framework provides container runtime environments for common collector needs, including PowerShell, Azure (Az) PowerShell, and VMware PowerCLI, on supported Ubuntu base images. Running a collector in a container gives it a clean, repeatable runtime.

A containerized collector reads its credentials from environment variables on the host or orchestrator, so no secrets are stored in the image.

Scheduling

Run a self-hosted collector on a schedule using the scheduler native to your platform:

• On Windows, use Task Scheduler to run the PowerShell script.
• On Linux, use cron.
• In a container platform, use the platform’s scheduled job mechanism.

Match the schedule to your billing cycle so that data is collected before validation and invoicing.

Troubleshooting

• Pending .rdy files: indicate data that was saved but not yet posted. Re-running the post step retries them.
• API responses: the module saves responses alongside the data files; review them to see why a submission was rejected.
• Verbose logging: run the collector with -VerboseOutput to see detailed diagnostic output.
• Authentication failures: confirm the Tenant Key, Source Collector Key, and Source Collector Secret match the values in the collector editor, and that the collector is set to Self Hosted.