performance - 하이브 - 이 MapReduce에서 최종 축소 단계가 극단적으로 느린 이유는 무엇입니까?(HiveQL, HDFS MapReduce)



하둡 용도 (1)

최종 감속기가 조인이면 조인 키의 비뚤어 짐처럼 보입니다. 우선 두 가지를 확인하십시오.

b.f1 조인 키에 중복이 없는지 확인하십시오.

select b.f1, count(*) cnt from B b 
 group by b.f1 
having count(*)>1 order by cnt desc;

a.f1의 분포를 확인하십시오.

select a.f1, count(*) cnt from A a
 group by a.f1  
order by cnt desc
limit 10;

이 쿼리는 비뚤어진 키를 보여줍니다.

왜곡 (동일한 값을 가진 행이 너무 많음)이있는 경우 왜곡 된 키를 별도로 조인하면 union all을 사용하십시오.

SELECT a.f1, f2, ..., fn
  FROM ( select * from A where f1 = skewed_value) as a --skewed
  LEFT JOIN B as b
  ON a.f1 = b.f1
WHERE {PARTITION_FILTER}
UNION ALL
SELECT a.f1, f2, ..., fn
  FROM ( select * from A where f1 != skewed_value) as a --all other
  LEFT JOIN B as b
  ON a.f1 = b.f1
WHERE {PARTITION_FILTER}

마지막으로 왜곡 및 중복 문제가없는 경우 감소 기 병렬 처리를 늘리십시오. 감속기 구성 당 현재 바이트 가져 오기

set hive.exec.reducers.bytes.per.reducer;

전형적으로 이것은 대략 1G에 대한 어떤 값을 반환 할 것이다. 2로 나누고 쿼리 전에 새 값을 설정하고 얼마나 많은 감속기가 시작되고 성능이되는지 확인하십시오. 성공 기준은 더 많은 감속기가 시작되었고 성능이 개선되었습니다.

set hive.exec.reducers.bytes.per.reducer=67108864;

감속기 당 바이트 수가 적을수록 감속기가 더 많이 시작되어 병렬 처리가 증가합니다.

업데이트 : 지도 조인을 사용하도록 설정 하십시오. 두 번째 테이블이 메모리에 들어가기에 충분히 작다면 mapjoin은 감속기 없이도 작동하며 감속기의 기울이기에는 문제가 없습니다.

mapjoin을 사용하도록 설정하는 방법 : https://stackoverflow.com/a/49154414/2700344

배경 정보 :

저는 Dataiku DSS, HDFS 및 파티션 된 데이터 세트로 작업하고 있습니다. 하나의 매우 큰 파티션 된 데이터 세트, 다른 하나는 작은 (~ 250 개의 행, 2 개의 열) 비 파티션 된 데이터 세트 인 두 개의 입력 데이터 세트가있는 특정 작업 (하이브 쿼리)이 있습니다. 분할 된 테이블 A와 분할되지 않은 테이블 B를 호출 해 봅시다.

의문:

쿼리의 형식은 다음과 같습니다.

SELECT a.f1, f2, ..., fn
  FROM A as a
  LEFT JOIN B as b
  ON a.f1 = b.f1
WHERE {PARTITION_FILTER}

다음은 MapReduce 작업의 현재 출력입니다 (이 작업이 아직 실행 중임을 명심하십시오).

[09:05:53] [INFO] [dku.utils]  - INFO  : Total jobs = 4
[09:05:53] [INFO] [dku.utils]  - INFO  : Starting task [Stage-10:CONDITIONAL] in serial mode
[09:05:53] [INFO] [dku.utils]  - INFO  : Stage-11 is filtered out by condition resolver.
[09:05:53] [INFO] [dku.utils]  - INFO  : Stage-1 is selected by condition resolver.
[09:05:53] [INFO] [dku.utils]  - INFO  : Launching Job 1 out of 4
[09:05:53] [INFO] [dku.utils]  - INFO  : Starting task [Stage-1:MAPRED] in serial mode
[09:05:53] [INFO] [dku.utils]  - INFO  : Number of reduce tasks not specified. Estimated from input data size: 307
[09:05:53] [INFO] [dku.utils]  - INFO  : In order to change the average load for a reducer (in bytes):
[09:05:53] [INFO] [dku.utils]  - INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
[09:05:53] [INFO] [dku.utils]  - INFO  : In order to limit the maximum number of reducers:
[09:05:53] [INFO] [dku.utils]  - INFO  :   set hive.exec.reducers.max=<number>
[09:05:53] [INFO] [dku.utils]  - INFO  : In order to set a constant number of reducers:
[09:05:53] [INFO] [dku.utils]  - INFO  :   set mapreduce.job.reduces=<number>
[09:05:53] [INFO] [dku.utils]  - INFO  : number of splits:75
[09:05:53] [INFO] [dku.utils]  - INFO  : Submitting tokens for job: job_1529508387832_10211
[09:05:53] [INFO] [dip.hiveserver2.log.sniffer]  - sniffed applicationId => application_1529508387832_10211/
[09:05:53] [INFO] [dku.utils]  - INFO  : Kill Command = /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/bin/hadoop job  -kill job_1529508387832_10211
[09:05:53] [INFO] [dku.utils]  - INFO  : Hadoop job information for Stage-1: number of mappers: 75; number of reducers: 307
[09:05:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:05:47,749 Stage-1 map = 0%,  reduce = 0%
[09:06:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:06:48,444 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 278.18 sec
[09:06:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:06:51,682 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 373.35 sec
[09:07:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:00,159 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 501.46 sec
[09:07:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:02,235 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 539.6 sec
[09:07:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:07,472 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 1389.81 sec
[09:07:13] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:10,605 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 2172.52 sec
[09:07:13] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:12,574 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 2577.81 sec
[09:07:13] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:13,604 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 2865.44 sec
[09:07:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:15,747 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU 3110.21 sec
[09:07:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:19,898 Stage-1 map = 10%,  reduce = 0%, Cumulative CPU 4080.2 sec
[09:07:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:21,988 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU 4522.48 sec
[09:07:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:23,015 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 4755.96 sec
[09:07:33] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:29,335 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 5710.85 sec
[09:07:33] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:31,407 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 5948.34 sec
[09:07:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:34,555 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU 6399.6 sec
[09:07:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:37,663 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 6811.22 sec
[09:07:43] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:38,695 Stage-1 map = 19%,  reduce = 0%, Cumulative CPU 7087.68 sec
[09:07:43] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:39,729 Stage-1 map = 20%,  reduce = 0%, Cumulative CPU 7288.22 sec
[09:07:43] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:40,769 Stage-1 map = 22%,  reduce = 0%, Cumulative CPU 7520.54 sec
[09:07:43] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:41,903 Stage-1 map = 24%,  reduce = 0%, Cumulative CPU 7771.37 sec
[09:07:43] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:42,930 Stage-1 map = 25%,  reduce = 0%, Cumulative CPU 7936.9 sec
[09:07:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:45,035 Stage-1 map = 27%,  reduce = 0%, Cumulative CPU 8254.78 sec
[09:07:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:46,075 Stage-1 map = 29%,  reduce = 0%, Cumulative CPU 8428.35 sec
[09:07:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:47,111 Stage-1 map = 30%,  reduce = 0%, Cumulative CPU 8661.23 sec
[09:07:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:48,153 Stage-1 map = 31%,  reduce = 0%, Cumulative CPU 8834.37 sec
[09:07:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:49,193 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 8983.68 sec
[09:07:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:50,227 Stage-1 map = 35%,  reduce = 0%, Cumulative CPU 9149.94 sec
[09:07:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:51,263 Stage-1 map = 37%,  reduce = 0%, Cumulative CPU 9268.9 sec
[09:07:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:52,301 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 9415.86 sec
[09:07:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:53,352 Stage-1 map = 39%,  reduce = 0%, Cumulative CPU 9540.63 sec
[09:07:58] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:54,381 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU 9711.54 sec
[09:07:58] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:55,421 Stage-1 map = 43%,  reduce = 0%, Cumulative CPU 9823.52 sec
[09:07:58] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:56,453 Stage-1 map = 46%,  reduce = 0%, Cumulative CPU 10010.83 sec
[09:07:58] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:57,492 Stage-1 map = 49%,  reduce = 0%, Cumulative CPU 10081.9 sec
[09:07:58] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:58,532 Stage-1 map = 53%,  reduce = 0%, Cumulative CPU 10230.13 sec
[09:08:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:07:59,576 Stage-1 map = 56%,  reduce = 0%, Cumulative CPU 10392.61 sec
[09:08:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:00,607 Stage-1 map = 58%,  reduce = 0%, Cumulative CPU 10483.38 sec
[09:08:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:01,649 Stage-1 map = 63%,  reduce = 0%, Cumulative CPU 10618.16 sec
[09:08:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:02,672 Stage-1 map = 66%,  reduce = 0%, Cumulative CPU 10684.82 sec
[09:08:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:03,695 Stage-1 map = 67%,  reduce = 0%, Cumulative CPU 10701.95 sec
[09:08:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:04,720 Stage-1 map = 70%,  reduce = 0%, Cumulative CPU 10767.21 sec
[09:08:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:05,750 Stage-1 map = 71%,  reduce = 0%, Cumulative CPU 10849.92 sec
[09:08:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:06,780 Stage-1 map = 73%,  reduce = 0%, Cumulative CPU 10924.45 sec
[09:08:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:07,902 Stage-1 map = 76%,  reduce = 0%, Cumulative CPU 11000.21 sec
[09:08:13] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:09,965 Stage-1 map = 77%,  reduce = 0%, Cumulative CPU 11013.58 sec
[09:08:13] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:10,991 Stage-1 map = 78%,  reduce = 0%, Cumulative CPU 11057.76 sec
[09:08:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:14,216 Stage-1 map = 80%,  reduce = 0%, Cumulative CPU 11157.51 sec
[09:08:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:17,362 Stage-1 map = 82%,  reduce = 0%, Cumulative CPU 11392.85 sec
[09:08:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:20,460 Stage-1 map = 83%,  reduce = 0%, Cumulative CPU 11610.7 sec
[09:08:28] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:26,106 Stage-1 map = 84%,  reduce = 0%, Cumulative CPU 11781.65 sec
[09:08:28] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:28,163 Stage-1 map = 85%,  reduce = 0%, Cumulative CPU 11788.58 sec
[09:08:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:35,410 Stage-1 map = 86%,  reduce = 0%, Cumulative CPU 12167.24 sec
[09:08:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:38,540 Stage-1 map = 86%,  reduce = 1%, Cumulative CPU 12317.09 sec
[09:08:43] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:39,583 Stage-1 map = 86%,  reduce = 2%, Cumulative CPU 12329.8 sec
[09:08:43] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:40,631 Stage-1 map = 86%,  reduce = 3%, Cumulative CPU 12333.61 sec
[09:08:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:47,927 Stage-1 map = 87%,  reduce = 3%, Cumulative CPU 12651.77 sec
[09:08:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:48,970 Stage-1 map = 88%,  reduce = 4%, Cumulative CPU 12826.37 sec
[09:08:53] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:51,092 Stage-1 map = 88%,  reduce = 5%, Cumulative CPU 12857.19 sec
[09:08:58] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:54,214 Stage-1 map = 90%,  reduce = 5%, Cumulative CPU 13037.63 sec
[09:09:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:08:59,396 Stage-1 map = 91%,  reduce = 5%, Cumulative CPU 13117.71 sec
[09:09:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:00,440 Stage-1 map = 92%,  reduce = 5%, Cumulative CPU 13238.06 sec
[09:09:03] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:01,485 Stage-1 map = 93%,  reduce = 5%, Cumulative CPU 13249.8 sec
[09:09:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:05,660 Stage-1 map = 94%,  reduce = 5%, Cumulative CPU 13306.0 sec
[09:09:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:06,706 Stage-1 map = 97%,  reduce = 5%, Cumulative CPU 13393.5 sec
[09:09:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:07,751 Stage-1 map = 97%,  reduce = 6%, Cumulative CPU 13409.12 sec
[09:09:08] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:08,795 Stage-1 map = 98%,  reduce = 6%, Cumulative CPU 13433.07 sec
[09:09:13] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:09,835 Stage-1 map = 98%,  reduce = 8%, Cumulative CPU 13474.03 sec
[09:09:13] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:10,874 Stage-1 map = 98%,  reduce = 9%, Cumulative CPU 13484.64 sec
[09:09:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:14,004 Stage-1 map = 100%,  reduce = 11%, Cumulative CPU 13580.71 sec
[09:09:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:15,118 Stage-1 map = 100%,  reduce = 13%, Cumulative CPU 13619.15 sec
[09:09:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:16,160 Stage-1 map = 100%,  reduce = 16%, Cumulative CPU 13707.2 sec
[09:09:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:17,210 Stage-1 map = 100%,  reduce = 33%, Cumulative CPU 14456.75 sec
[09:09:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:18,258 Stage-1 map = 100%,  reduce = 39%, Cumulative CPU 14708.07 sec
[09:09:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:19,291 Stage-1 map = 100%,  reduce = 40%, Cumulative CPU 14768.29 sec
[09:09:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:20,329 Stage-1 map = 100%,  reduce = 41%, Cumulative CPU 14834.88 sec
[09:09:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:21,360 Stage-1 map = 100%,  reduce = 42%, Cumulative CPU 14902.4 sec
[09:09:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:22,399 Stage-1 map = 100%,  reduce = 45%, Cumulative CPU 15040.16 sec
[09:09:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:23,433 Stage-1 map = 100%,  reduce = 47%, Cumulative CPU 15165.58 sec
[09:09:28] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:28,627 Stage-1 map = 100%,  reduce = 63%, Cumulative CPU 15792.29 sec
[09:09:33] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:30,711 Stage-1 map = 100%,  reduce = 64%, Cumulative CPU 15889.21 sec
[09:09:33] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:31,753 Stage-1 map = 100%,  reduce = 65%, Cumulative CPU 15898.95 sec
[09:09:33] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:32,789 Stage-1 map = 100%,  reduce = 66%, Cumulative CPU 15927.26 sec
[09:09:33] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:33,822 Stage-1 map = 100%,  reduce = 71%, Cumulative CPU 16086.93 sec
[09:09:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:34,866 Stage-1 map = 100%,  reduce = 90%, Cumulative CPU 16711.13 sec
[09:09:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:35,907 Stage-1 map = 100%,  reduce = 93%, Cumulative CPU 16795.34 sec
[09:09:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:36,952 Stage-1 map = 100%,  reduce = 95%, Cumulative CPU 16881.47 sec
[09:09:38] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:37,995 Stage-1 map = 100%,  reduce = 96%, Cumulative CPU 16891.18 sec
[09:09:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:44,249 Stage-1 map = 100%,  reduce = 97%, Cumulative CPU 16958.21 sec
[09:09:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:45,292 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 17011.88 sec
[09:09:48] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:09:47,378 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 17055.07 sec
[09:10:49] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:10:47,421 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 17545.89 sec
[09:11:49] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:11:47,872 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 17764.45 sec
[09:12:49] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:12:48,287 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 18330.86 sec
[09:13:49] [INFO] [dku.utils]  - INFO  : 2018-06-27 09:13:48,855 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 19232.58 sec
...
...
...
...
...
[10:22:20] [INFO] [dku.utils]  - INFO  : 2018-06-27 10:22:17,545 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 26857.28 sec
[10:23:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 10:23:17,654 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 26974.96 sec
[10:24:18] [INFO] [dku.utils]  - INFO  : 2018-06-27 10:24:18,112 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 27081.0 sec
[10:25:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 10:25:18,964 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 27187.52 sec
[10:26:23] [INFO] [dku.utils]  - INFO  : 2018-06-27 10:26:19,404 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 27287.58 sec

보시다시피, 감축 단계의 최종 1 %는 꽤 오래 걸립니다 (지금까지 1 시간 32 분의 1 시간 27 분 경과).

왜 이런 일이 일어나고 있는지 명확하고 간결한 설명을주는 자료를 찾는 데 어려움을 겪고 있습니다. 필자는 MapReduce에 대한 기본적인 지식이 있지만 현재 HDFS 환경에서 MapReduce를 연구하는 중입니다. 필자가 읽은 바에 따르면 JOIN 수행으로 인해 감축 작업의 대부분이 하나의 감속기에 고정 될 가능성이 있습니다.

누가 무슨 일이 일어나고 있는지 높은 수준의 설명을 줄 수있는 좋은 자원의 방향으로 나를 가리킨 다음 하이브 쿼리를 변경하여 향후 작업에서이를 피할 수있는 방법에 대한 정보를 제공하면 크게 감사하겠습니다.





reduce