Обновите ClickHouse с 1.1.54231 до 19.6.2.11

Наконец, мы решили обновить наш производственный сервер ClickHouse версии 1.1.54231 до новой версии 19.6. Новая версия запустилась, все конфиги подобрал, данные на сервере тоже видит, но на Zookeeper новые данные забрать не может. В журнале ошибок

    Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: user_files_path (version 19.6.2.11 (official build))
    Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: format_schema_path (version 19.6.2.11 (official build))
    Include not found: clickhouse_remote_servers
    Include not found: clickhouse_compression
    Logging error to /var/log/clickhouse-server/clickhouse-server.log
    Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
    Logging error to console
    2019.05.16 17:06:10.801816 [ 1 ] {} <Error> Application: Listen [0.0.0.0]: 98: Net Exception: Address already in use: 0.0.0.0:8123  If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
    2019.05.16 17:06:10.859589 [ 25 ] {} <Error> analytics..inner.main_page_hits (StorageReplicatedMergeTree): Code: 40, e.displayText() = DB::Exception: Checksums of parts don't match: hash of uncompressed files doesn't match (version 19.6.2.11 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons: 
1. Using newer version of compression library after server update. 
2. Using another compression method. 
3. Non-deterministic compression algorithm (highly unlikely). 
4. Non-deterministic merge algorithm due to logical error in code. 
5. Data corruption in memory due to bug in code. 
6. Data corruption in memory due to hardware issue. 
7. Manual modification of source data after server startup. 
8. Manual modification of checksums stored in ZooKeeper. We will download merged part from replica to force byte-identical result.
    2019.05.16 17:06:10.874886 [ 23 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8., Stack trace:

    0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x767c3f6]
    1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x39294a2]
    2. /usr/bin/clickhouse-server() [0x6d7fc8c]
    3. /usr/bin/clickhouse-server(DB::DataPartsExchange::Fetcher::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, DB::ConnectionTimeouts const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x950) [0x6d830b0]
    4. /usr/bin/clickhouse-server() [0x6d2d7a4]
    5. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, unsigned long)+0x97f) [0x6d48d5f]
    6. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&)+0x7f2) [0x6d4b012]
    7. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&)+0x26b) [0x6d4bd5b]
    8. /usr/bin/clickhouse-server() [0x6d4c31f]
    9. /usr/bin/clickhouse-server(DB::ReplicatedMergeTreeQueue::processEntry(std::function<std::shared_ptr<zkutil::ZooKeeper> ()>, std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&, std::function<bool (std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>)+0x64) [0x6e6d324]
    10. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::queueTask()+0x189) [0x6d280d9]
    11. /usr/bin/clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x44e) [0x6d7da4e]
    12. /usr/bin/clickhouse-server() [0x6d7e4c9]
    13. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x1ab) [0x768238b]
    14. /usr/bin/clickhouse-server() [0xb5488bf]
    15. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f553fd076db]
    16. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f553f28688f]
     (version 19.6.2.11 (official build))
    2019.05.16 17:06:10.886495 [ 25 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8.,

Любые идеи?


person Sergey Ly    schedule 17.05.2019    source источник


Ответы (1)


Похоже, вы обновили только одну реплику из нескольких.

В реплицированных сценариях результаты фонового слияния перекрестно проверяются с помощью контрольных сумм. Если результат слияния двух реплик не является идентичной по байтам репликой, отправьте это сообщение в журналы и повторно загрузите проблемную часть из других реплик. После обновления других реплик проблема исчезнет.

person filimonov    schedule 25.05.2019
comment
Да спасибо, завтра попробуем обновить сразу весь кластер. - person Sergey Ly; 27.05.2019
comment
@SergeyLy, как все прошло? - person Ivan Blinkov; 03.06.2019