#!/usr/bin/perl -w

=head1 NAME

afs-audit-volumes

=head1 SYNOPSIS

afs-audit-volumes [--listvldb listvldb.out] [--min-replicas n]
  [--ignore-domain] [--quiet|-q] [--commands=file] [listvol.out ...]


=head1 OPTIONS

=over 8

=item B<--listvldb|-v> file

Output of the vos listvldb command.

=item B<--commands> file

Print suggested AFS commands to correct detected errors.

=item B<--min-replicas> n

Minimum number of recommended read-only volumes. Default is 1.

=item B<--ignore-domain>

Ingore the domain name from vldb entries to match the
server names in the vos listvol output. This is necessary when
host name resolution is not consistent.

=item B<--quiet|-q>

Less messages.

=item B<--help|-h>

Print syntax.

=back

=head1 DESCRIPTION

afs-audit-volumes compares the output of 'vos listvldb' and 'vos
listvol' from each fileserver, and checks for potential problems
and inconsistences.

By default, afs-audit-volumes will run 'vos listvldb' and from the
output of 'vos listvldb' determine which fileservers are present
in the cell. afs-audit-volumes will then run 'vos listvol' for
each fileserver found from the 'vos listvldb' output. 

afs-audit-volumes compares the information retrieved from the volume
location database to the information retrieved from each fileserver.

Optionally, you may run 'vos listvldb' and for each fileserver,
'vos listvol', saving the output of those commands to files, and
then run afs-audit-volumes by specifying the files as command line
arguments. 


=head2 Server Names

The server host names reported by vos are resolved by
ip to name look ups.  Depending on how names are resolved,
the server names may be fully qualified or may just contain
the host name.  In these cases, the --ignore-domain
can be used to attempt to ignore domain names.

For example, if the output of vos listvldb is of the form,

 my.vol
    RWrite: 1819084418    ROnly: 1819084419    Backup: 1819084420
    number of sites -> 3
       server afs1.mydomain.com partition /vicepj RW Site
       server afs1.mydomain.com partition /vicepj RO Site
       server afs2.mydomain.com partition /vicept RO Site

but the output of vos listvol is,

  ...
  Total number of volumes on server afs1 partition /vicepj: 3
  my.vol                      1819084418 RW    1836860 K On-line
  my.vol.readonly             1819084419 RO    1836860 K On-line
  my.vol.backup               1819084420 BK    1836860 K On-line
  ...
  Total number of volumes on server afs2 partition /vicept: 1
  my.vol.readonly             1819084419 RO    1836860 K On-line
  ...

then, the --ignore-domain option should be used to force the
script to ignore the mydomain.com in the vos listvldb entries.

=head2 Errors Reported

The types of errors reported:

=over 8

=item Attach errors

Volume was not attached as reported by vos listvol.

=item Duplicate read-only id

A duplicate read-only volume id was detected. If not corrected,
clients may see the wrong read-only data.

=item Duplicate read-only name

A duplicate read-only volume name was dectected.  If this volume
is not in the vldb, this error reduces the amount of free disk space
and the files in the old read-only volume are potentially out of date.

=item Misplaced read-only

A local read-only volume is not on the same partition as the
master read-write volume. This reduces the amount of free disk space.

=item VLDB missing read-write

A VLDB entry does not have a read-write volume. Changes to the read-only
volumes are not possible.

=item VLDB too many read-write

A VLDB entry has multiple read-write sites.

=item VLDB duplicate name

A duplicate volume name was detected in the VLDB.

=item VLDB duplicate id

A duplicate id was detected in the VLDB.

=item Not enough replicas

The recommended number of read-only volumes are not present for
a replicated volume. The minimun number is specified with the
--min-replicas option. The default is 1.

=item Volume location inconsistencies

A volume in the VLDB was not found in the correct
file server partition.

=item Volume type inconsistencies

A volume in the VLDB has a different type on the file
server partiton.

=item Volume name inconsistencies

A volumen in the VLDB has a different name on the file
server partition.

=item RW Volume missing in vldb

No vldb entry was found for a read-write volume
on a fileserver.

=item RO Volume missing in vldb

No vldb entry was found for a read-only volume
on a fileserver.

=item Wrong location in vldb

The wrong server or partition was found in the
vldb for a volume on a fileserver.

=back

=head1 EXAMPLES

Run afs-audit-volumes without options to retrieve and check the
volume information:

    $ afs-audit-volumes
    Found servers: afs01.sinenomine.net, afs03.sinenomine.net, afs04.sinenomine.net
    Getting volume list from server afs01.sinenomine.net...
    Getting volume list from server afs03.sinenomine.net...
    Getting volume list from server afs04.sinenomine.net...
    Searching for duplicate read-only volume ids...
    Searching for duplicate read-only volume names...
    Searching for misplaced read-only volumes...
    Searching for duplicates in vldb...
    Checking for sufficient number of replicas...
    Searching for vldb/fileserver inconsistencies...
    Searching for fileserver/vldb rw volume inconsistencies...
    Searching for fileserver/vldb ro volumes inconsistencies...
    
    SUMMARY
    =======
    Attach errors                     :  0
    Duplicate read-only id            :  0
    Duplicate read-only name          :  0
    Misplaced read-only               :  0
    VLDB missing read-write           :  0
    VLDB too many read-write          :  0
    VLDB duplicate name               :  0
    VLDB duplicate id                 :  0
    Not enough replicas               :  0
    Volume location inconsistencies   :  0
    Volume type inconsistencies       :  0
    Volume name inconsistencies       :  0
    RW Volume missing in vldb         :  0
    RO Volume missing in vldb         :  0
    Wrong location in vldb            :  0


Gather the volume information before running afs-audit-volumes. This
allows you to separation the gathering of the volume data from the
analysis.


  # Retrieve the current vldb.
  vos listvldb > mycell.listvldb

  # Retrieve the file server volume lists.
  for s in `vos listaddrs`
  do
    vos listvol $s >$s.listvol
  done

  # Run the volume audit.
  afs-audit-volume --listvldb mycell.listvldb *.listvol


=head1 COPYRIGHT

Copyright (c) 2008, Sine Nomine Associates
All rights reserved.

This program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

=cut

use Getopt::Long;
use Pod::Usage;

my $opt_commands = '';
my $opt_listvldb = '';
my $opt_ignore_domain = 0;
my $opt_min_replicas = 1;
my $opt_quiet = 0;
GetOptions(
    "listvldb|v=s"   => \$opt_listvldb,
    "ignore-domain"  => \$opt_ignore_domain,
    "min-replicas=i" => \$opt_min_replicas,
    "commands|c=s"   => \$opt_commands,
    "quiet|q"        => \$opt_quiet,
    "help|h|?"       => \$opt_help,
) or pod2usage(2);
pod2usage(1) if $opt_help;


# vos listvol fields, with server/partition prepended
use constant SERVER     => 0;
use constant PARTITION  => 1;
use constant NAME       => 2;
use constant ID         => 3;
use constant TYPE       => 4;
use constant SIZE       => 5;
use constant UNITS      => 6;
use constant STATUS     => 7;

my $summary = {
  AUDIT_ATTACH_ERROR => 0,
  AUDIT_DUP_RO_ID => 0,
  AUDIT_DUP_RO_NAME => 0,
  AUDIT_ERRANT_RO => 0,
  AUDIT_VLDB_MISSING_RW => 0,
  AUDIT_VLDB_TOO_MANY_RW => 0,
  AUDIT_VLDB_DUP_NAME => 0,
  AUDIT_VLDB_DUP_ID => 0,
  AUDIT_MIN_REPLICAS => 0,
  AUDIT_VOL_LOCATION => 0,
  AUDIT_VOL_TYPE => 0,
  AUDIT_VOL_NAME => 0,
  AUDIT_RW_NOT_IN_VLDB =>0,
  AUDIT_RO_NOT_IN_VLDB =>0,
  AUDIT_WRONG_LOCATION_IN_VLDB=>0,
};


my %g_partition_check = ();
my $g_vldb = [];
my @g_vl_servers = ();
my @g_vol_servers = ();

if ($opt_commands) {
    open(CMDS, "> $opt_commands") or die "error: Failed to open $opt_commands for writing. $!\n";
    print "Writing commands to $opt_commands\n";
}


read_listvldb($opt_listvldb);
unless($opt_quiet) {
    print "Found servers: ", join(", ", @g_vl_servers), "\n";
}

if (scalar(@ARGV) == 0) {
    foreach my $server (@g_vl_servers) {
        read_listvol('server' => $server);
    }
}
else {
    foreach my $filename (@ARGV) {
        read_listvol('filename' => $filename);
    }
}


audit_duplicate_ro_id();
audit_duplicate_ro_name();
audit_errant_ro();
audit_vldb_one_readwrite();
audit_vldb_under_replicated();
audit_vldb_to_partition_consistency();
audit_fs_to_vldb_consistency_rw();
audit_fs_to_vldb_consistency_ro();

print_summary();


#
# Read the output of vos listvol for one or more servers. Store
# into the $servers data structure.
#
sub read_listvol
{
    my %arg = @_;
    my $server = undef;
    my $partition = undef;
    my @servers;

    if ($arg{'filename'}) {
        my $filename = $arg{'filename'};
        print "Reading volume list $filename...\n" unless $opt_quiet;
        open(LISTVOL, "< $filename") or die "error: Failed to open intput file $filename.\n";
    }
    elsif ($arg{'server'}) {
        my $server = $arg{'server'};
        print "Getting volume list from server $server...\n" unless $opt_quiet;
        my $vos = "vos listvol -server $server -noauth";
        open(LISTVOL, "$vos |") or die "error: Failed to run $vos: $!\n";
    }
    else {
        die "error: read_listvol invalid argument.\n";
    }

    while (<LISTVOL>) {
        chomp;
        next if /^\s*$/;                  # skip blank lines
        next if /^Total volumes/;         # skip summary lines
        if (/Could not attach volume/) {  # print not attach as a warning.
            $summary->{AUDIT_ATTACH_ERROR}++;
            unless ($opt_quiet) {
                s/\*//g;
                s/^\s*//;
                print "warning: $_\n";
            }
        }
        elsif (/Total number of volumes on server (.*) partition \/vicep(.)/) {
            my $server_name = $1;
            my $partition_name = $2;

            if ($opt_ignore_domain) {
                $server_name =~ s/\..*$//;
            }

            # avoid duplicate vos listvol.
            my $key = "$server_name:$partition_name";
            if (defined $g_partition_check{$key}) {
                die "error: Duplicate vos listvol input: $key\n";
            }
            $g_partition_check{$key} = 1;

            # save previous partitions
            if (defined $partition) {
                push(@{$server->{PARTITIONS}}, $partition);
            }
            $partition = {
                NAME => $partition_name,
                RW => [],
                RO => [],
            };

            # save previous server list
            if (defined $server && ($server->{NAME} ne $server_name)) {
                push(@g_vol_servers, $server);
                $server = undef;
            }
            unless (defined $server) {
                $server = {
                    NAME => $server_name,
                    PARTITIONS => [],
                }
            }
        }
        else {
            my @volume = ($server->{NAME}, $partition->{NAME}, split);
            if (scalar @volume != (STATUS+1)) {
                die "error: wrong number of listvol fields.\n$_\n";
            }

            if ($volume[TYPE] eq 'RW') {
                push(@{$partition->{RW}}, \@volume);
            }
            if ($volume[TYPE] eq 'RO') {
                push(@{$partition->{RO}}, \@volume);
            }
        }
    }

    if (defined $partition) {
        push(@{$server->{PARTITIONS}}, $partition);
    }
    if (defined $server) {
        push(@g_vol_servers, $server);
    }
}

#
# Read the vos listvldb output.
#
sub read_listvldb
{
    my $filename = shift;

    my %servers;
    my $name;
    my $rw;
    my $ro;
    my $bk;
    my $rc;
    my $entry;

    if ($filename) {
        open(LISTVLDB, "< $filename") or die "error: Failed to open intput file $filename: $!\n";
    }
    else {
        my $vos = "vos listvldb -noauth";
        open(LISTVLDB, "$vos |") or die "error: Failed to run $vos: $!\n";
    }

    while (<LISTVLDB>) {
        chomp;
        next if /^\s*$/;    # skip blank lines
        next if /^VLDB entries/;
        next if /^Total entries:/;
        next if /number of sites/;

        if (/^([\w\-\._]+)/) {  # volume name
            $name = $1;
            $rw = undef;
            $ro = undef;
            $bk = undef;
            $rc = undef;
            if (defined $entry) {
                push(@{$g_vldb}, $entry);
            }
            $entry = {
                NAME => $name,
                ID => 0,
                STATUS => 0,
                IDS => {},
                SITES => {
                    RW => [],
                    RO => [],
                },
            };
            next;
        }
        if (/RWrite:\s+(\d+)/) {
            $rw = $1;
            $entry->{IDS}->{RW} = $rw;
            $entry->{ID} = $rw;
        }
        if (/ROnly:\s+(\d+)/) {
            $ro = $1;
            $entry->{IDS}->{RO} = $ro;
        }
        if (/Backup:\s+(\d+)/) {
            $bk = $1;
            $entry->{IDS}->{BK} = $bk;
        }
        if (/RClone:\s+(\d+)/) {
            $rc = $1;
            $entry->{IDS}->{RC} = $rc;
        }
        if (/^\s+server (.*) partition \/vicep(..?) (..) Site/) {
            unless ($entry) {
                die "error: Missing volume name, $_\n";
            }
            my $server = $1;
            my $partition = $2;
            my $type = $3;
            my $status;
            my $id;
            my $sites;

            if ($opt_ignore_domain) {
                $server =~ s/\..*$//;
            }
            $servers{$server} = 1;

            if (/-- New release/) {
                $status = 'new';
                $entry->{STATUS} = 1;
            }
            elsif (/-- Old release/) {
                $status = 'old';
                $entry->{STATUS} = 1;
            }
            elsif (/-- Not released/) {
                $status = 'not';
                $entry->{STATUS} = 1;
            }
            else {
                $status = 'ok';
            }

            if ($type eq 'RW') {
                $id = $rw;
                $sites = $entry->{SITES}->{RW};
            }
            elsif ($type eq 'RO') {
                $id = $ro;
                $sites = $entry->{SITES}->{RO};
            }
            else {
                die "error: Unexpected type in vldb: $_\n";
            }

            die "fatal: vldb missing server" unless $server;
            die "fatal: vldb missing partition" unless $partition;
            die "fatal: vldb missing name" unless $name;
            if ($status ne 'not') { # addsite before release will not have an id yet.
                die "fatal: vldb missing id. $status" unless $id;
            }
            die "fatal: vldb missing type" unless $type;

            my $site = [$server, $partition, $name, $id, $type, '', '', $status];
            push(@{$sites}, $site);
        }
    }
    if (defined $entry) {
        push(@{$g_vldb}, $entry);
    }
    close LISTVLDB;

    @g_vl_servers = sort(keys(%servers));
    return;
}

#
# Check for duplicate ro ids on each server.
#
sub audit_duplicate_ro_id
{
    print "Searching for duplicate read-only volume ids...\n" unless $opt_quiet;
    foreach my $s (@g_vol_servers) {
        my %hash;
        foreach my $p (@{$s->{PARTITIONS}}) {
            foreach my $v (@{$p->{RO}}) {
                $key = $v->[ID];
                if (!defined $hash{$key}) {
                    $hash{$key} = 1;
                }
                else {
                    $summary->{AUDIT_DUP_RO_ID}++;
                    unless ($opt_quiet) {
                        print "error: Duplicate read-only volume id: $v->[SERVER] $v->[ID]\n";
                    }
                }
            }
        }
    }
}

#
# Check for duplicate ro volume names.
#
sub audit_duplicate_ro_name
{
    print "Searching for duplicate read-only volume names...\n" unless $opt_quiet;
    foreach my $s (@g_vol_servers) {
        my %hash;
        foreach my $p (@{$s->{PARTITIONS}}) {
            foreach my $v (@{$p->{RO}}) {
                $key = $v->[NAME];
                if (!defined $hash{$key}) {
                    $hash{$key} = 1;
                }
                else {
                    $summary->{AUDIT_DUP_RO_NAME}++;
                    unless ($opt_quiet) {
                        print "warning: Duplicate read-only volume name: $v->[SERVER] $v->[NAME]\n";
                    }
                }
            }
        }
    }
}

#
# Search for local read-only volumes that are not on the
# same partition as the read-write.
#
sub audit_errant_ro
{
    print "Searching for misplaced read-only volumes...\n" unless $opt_quiet;
    foreach my $s (@g_vol_servers) {
        foreach my $p (@{$s->{PARTITIONS}}) {
            foreach my $rw (@{$p->{RW}}) {
                find_errant_ro($s, $rw);
            }
        }
    }
}

#
# For a given rw volume, search find local ro volumes on the
# same server but on a different partition.
#
sub find_errant_ro
{
    my $s = shift;
    my $rw = shift;

    my $ro_id = find_ro_id($rw->[ID]);
    if ($ro_id) {
        foreach my $p (@{$s->{PARTITIONS}}) {
            next if ($p->{NAME} eq $rw->[PARTITION]);
            foreach my $ro (@{$p->{RO}}) {
                if ($ro->[ID] == $ro_id) {
                    $summary->{AUDIT_ERRANT_RO}++;
                    print "warning: Wrong partition: $ro->[NAME] ".
                          "on server $ro->[SERVER] $ro->[PARTITION] ".
                          "should be on $rw->[PARTITION]\n";
                    if ($opt_commands) {
                        print_command_errant_ro($rw, $ro);
                    }
                }
            }
        }
    }
}

#
# For a given rw volume id, find the corresponding ro id, if one.
#
sub find_ro_id
{
    my $rw = shift;
    foreach my $entry (@{$g_vldb}) {
        my $id = $entry->{ID};
        if ($id == $rw) {
           return $entry->{IDS}->{RO};
        }
    }
    return undef;
}

#
# Print the vos commands needed to fix an errant ro.
#
sub print_command_errant_ro
{
    my $rw = shift;
    my $ro = shift;

    print CMDS "# Move errant ro $ro->[SERVER] $ro->[NAME] $ro->[PARTITION] to partition $rw->[PARTITION]\n";
    print CMDS "vos remove -server $ro->[SERVER] -partition $ro->[PARTITION] -id $ro->[ID] -verbose && \\\n";
    print CMDS "  vos addsite -server $rw->[SERVER] -partition $rw->[PARTITION] -id $rw->[ID] -verbose && \\\n";
    print CMDS "  vos release -id $rw->[ID] -verbose\n\n";
}

#
# Verify each vlbd entry has one and only one unique read-write volumes.
#
sub audit_vldb_one_readwrite
{
    my %hash_name;
    my %hash_id;

    print "Searching for duplicates in vldb...\n" unless $opt_quiet;
    foreach $entry (@{$g_vldb}) {
        my $rw = $entry->{SITES}->{RW};
        my $name = $entry->{NAME};
        my $id = $entry->{ID};
        my $num_rw = scalar @{$rw};
        if ($num_rw == 0) {
            $summary->{AUDIT_VLDB_MISSING_RW}++;
            print "error: vldb entry is missing read-write volume: $entry->{NAME}\n";
        }
        elsif ($num_rw > 1) {
            $summary->{AUDIT_VLDB_TOO_MANY_RW}++;
            print "error: vldb entry has too many read-write entries: $entry->{NAME}\n";
        }
        if (defined $hash_name{$id}) {
            $summary->{AUDIT_VLDB_DUP_NAME}++;
            print "error: vldb repeated name: $entry->{NAME}\n";
        }
        if (defined $hash_id{$id}) {
            $summary->{AUDIT_VLDB_DUP_ID}++;
            print "error: vldb repeated id: $entry->{NAME}\n";
        }
    }
}

#
# Check for a minumum number of replicas if replicated.
#
sub audit_vldb_under_replicated
{
    my %hash_name;
    my %hash_id;

    print "Checking for sufficient number of replicas...\n" unless $opt_quiet;
    foreach $entry (@{$g_vldb}) {
        my $ro = $entry->{SITES}->{RO};
        my $name = $entry->{NAME};
        my $num_ro = scalar @{$ro};
        if ($num_ro > 0 && $num_ro < $opt_min_replicas) {
            $summary->{AUDIT_MIN_REPLICAS}++;
            print "warning: only $num_ro replicas: $entry->{NAME}\n";
        }
    }
}

#
# Cross check the listvldb with the listvol.
#
sub audit_vldb_to_partition_consistency
{
    print "Searching for vldb/fileserver inconsistencies...\n" unless $opt_quiet;
    foreach $entry (@{$g_vldb}) {
        my $name = $entry->{NAME};
        my $id = $entry->{ID};
        foreach my $site (@{$entry->{SITES}->{RW}}, @{$entry->{SITES}->{RO}}) {
            next if $site->[STATUS] eq 'not';  # skip not released entries
            my $sname = $name . ($site->[TYPE] eq 'RO' ? '.readonly' : '');

            my $v = find_volume_in_partition($site->[ID], $site->[SERVER], $site->[PARTITION], $site->[TYPE]);
            if (!defined $v) {
                $summary->{AUDIT_VOL_LOCATION}++;
                print "error: volume location inconsistency, $sname ($id), $site->[SERVER] $site->[PARTITION]\n";
            }
            else {
                if ($v->[TYPE] ne $site->[TYPE]) {
                    $summary->{AUDIT_VOL_TYPE}++;
                    print "error: volume type inconsistency, $sname ($id), $site->[SERVER] $site->[PARTITION]\n";
                }
                if ($v->[NAME] ne $sname) {
                    $summary->{AUDIT_VOL_NAME}++;
                    print "error: volume name inconsistency, $sname ($id), $site->[SERVER] $site->[PARTITION]\n";
                }
            }
        }
    }
}

sub audit_fs_to_vldb_consistency_rw
{
    print "Searching for fileserver/vldb rw volume inconsistencies...\n" unless $opt_quiet;
    my $volumes = {};
    foreach $entry (@{$g_vldb}) {
        my $rwlist = $entry->{SITES}->{RW};
        if (scalar @{$rwlist} == 0) {
            print "warning: vldb entry is missing rw site";
            print ", volume ", $entry->{NAME} if $entry->{NAME};
            print "\n";
        }
        else {
            if (scalar @{$rwlist} > 1) {
                print "warning: vldb entry has multiple rw sites";
                print ", volume ", $entry->{NAME} if $entry->{NAME};
                print "\n";
            }
            $rw = $rwlist->[0];
            $id = $rw->[ID];
            if (defined $volumes->{$id}) {
                print "warning: duplicate rw id in vldb";
                print ", volume ", $entry->{NAME} if $entry->{NAME};
                print "\n";
            }
            else {
                $volumes->{$id} = $rw;
            }
        }
    }

    foreach my $s (@g_vol_servers) {
        my %hash;
        foreach my $p (@{$s->{PARTITIONS}}) {
            foreach my $rw (@{$p->{RW}}) {
                my $v = $volumes->{$rw->[ID]};
                if (!defined $v) {
                    $summary->{AUDIT_RW_NOT_IN_VLDB}++;
                    print "warning: rw volume not found in vldb: ".
                          "$rw->[ID] $rw->[SERVER] $rw->[PARTITION]\n";
                }
                else {
                    if ($v->[SERVER] ne $rw->[SERVER]) {
                        $summary->{AUDIT_WRONG_LOCATION_IN_VLDB}++;
                        print "warning: wrong server name in vldb: ".
                               "$rw->[ID] found: $v->[SERVER] ".
                               "expected: $rw->[SERVER]\n";
                    }
                    if ($v->[PARTITION] ne $rw->[PARTITION]) {
                        $summary->{AUDIT_WRONG_LOCATION_IN_VLDB}++;
                        print "warning: wrong partition in vldb: ".
                               "$rw->[ID] found: $v->[PARTITION] ".
                               "expected: $rw->[PARTITION]\n";
                    }
                }
            }
        }
    }
}

sub audit_fs_to_vldb_consistency_ro
{
    print "Searching for fileserver/vldb ro volumes inconsistencies...\n" unless $opt_quiet;
    my $volumes = {};
    foreach $entry (@{$g_vldb}) {
        my $rolist = $entry->{SITES}->{RO};
        foreach my $ro (@{$rolist}) {
            if (defined($ro->[ID]) && defined($ro->[SERVER])) {
                my $key = $ro->[ID] . "\0" . $ro->[SERVER];
                if (defined ($volumes->{$key})) {
                    print "warning: duplicate ro id/server in vldb ";
                    print " $ro->[ID] $ro->[SERVER]\n";
                }
                else {
                    $volumes->{$key} = $ro;
                }
            }
        }
    }

    foreach my $s (@g_vol_servers) {
        my %hash;
        foreach my $p (@{$s->{PARTITIONS}}) {
            foreach my $ro (@{$p->{RO}}) {
                my $key = $ro->[ID] ."\0".  $ro->[SERVER];
                my $v = $volumes->{$key};
                if (!defined $v) {
                    $summary->{AUDIT_RO_NOT_IN_VLDB}++;
                    print "warning: ro volume not found in vldb: ".
                          "$ro->[ID] $ro->[SERVER] $ro->[PARTITION]\n";
                }
                else {
                    if ($v->[PARTITION] ne $ro->[PARTITION]) {
                        $summary->{AUDIT_WRONG_LOCATION_IN_VLDB}++;
                        print "warning: wrong partition in vldb: ".
                               "$ro->[ID] found: $v->[PARTITION] ".
                               "expected: $ro->[PARTITION]\n";
                    }
                }
            }
        }
    }
}

sub find_volume_in_partition
{
    my ($id, $server, $partition, $type) = @_;
    my $p = find_partition($server, $partition);
    if (defined $p) {
        foreach my $v (@{$p->{$type}}) {
            if ($v->[ID] eq $id) {
                return $v;
            }
        }
    }
    return undef;
}

sub find_partition
{
    my $server = shift;
    my $partition = shift;

    foreach my $s (@g_vol_servers) {
        if ($s->{NAME} eq $server) {
            foreach my $p (@{$s->{PARTITIONS}}) {
                if ($p->{NAME} eq $partition) {
                    return $p;
                }
            }
        }
    }
    return undef;
}

#
# Count total number of errors.
#
sub summary_total
{
    my $total = 0;
    foreach my $s (keys(%{$summary})) {
        $total += $summary->{$s};
    }
    return $total;
}

#
# Print a summary of the errors found.
#
sub print_summary
{
    print "\n" unless $opt_quiet;
    print "Summary\n" unless $opt_quiet;
    print "=======\n" unless $opt_quiet;
    print "Attach errors                   =  $summary->{AUDIT_ATTACH_ERROR}\n";
    print "Duplicate read-only id          =  $summary->{AUDIT_DUP_RO_ID}\n";
    print "Duplicate read-only name        =  $summary->{AUDIT_DUP_RO_NAME}\n";
    print "Misplaced read-only             =  $summary->{AUDIT_ERRANT_RO}\n";
    print "VLDB missing read-write         =  $summary->{AUDIT_VLDB_MISSING_RW}\n";
    print "VLDB too many read-write        =  $summary->{AUDIT_VLDB_TOO_MANY_RW}\n";
    print "VLDB duplicate name             =  $summary->{AUDIT_VLDB_DUP_NAME}\n";
    print "VLDB duplicate id               =  $summary->{AUDIT_VLDB_DUP_ID}\n";
    print "Not enough replicas             =  $summary->{AUDIT_MIN_REPLICAS}\n";
    print "Volume location inconsistencies =  $summary->{AUDIT_VOL_LOCATION}\n";
    print "Volume type inconsistencies     =  $summary->{AUDIT_VOL_TYPE}\n";
    print "Volume name inconsistencies     =  $summary->{AUDIT_VOL_NAME}\n";
    print "RW Volume missing in vldb       =  $summary->{AUDIT_RW_NOT_IN_VLDB}\n";
    print "RO Volume missing in vldb       =  $summary->{AUDIT_RO_NOT_IN_VLDB}\n";
    print "Wrong location in vldb          =  $summary->{AUDIT_WRONG_LOCATION_IN_VLDB}\n";
    print "TOTAL                           =  ". summary_total() ."\n";
}

